Extramissive spatial imaging digital eye glass apparatuses, methods and systems for virtual or augmediated vision, manipulation, creation, or interaction with objects, materials, or other entities

ABSTRACT

A sensing and display apparatus, comprising: a first phenomenon interface configured to operatively interface with a first augmediated-reality space, and a second phenomenon interface configured to operatively interface with a second augmediated-reality space, is implemented as an extramissive spatial imaging digital eye glass.

PRIORITY CLAIMS AND CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/156,798 titled “EXTRAMISSIVE SPATIAL IMAGING DIGITAL EYE GLASSAPPARATUSES, METHODS AND SYSTEMS FOR VIRTUAL OR AUGMEDIATED VISION,MANIPULATION, CREATION, OR INTERACTION WITH OBJECTS, MATERIALS, OR OTHERENTITIES” filed Oct. 10, 2018, which is a continuation of U.S. patentapplication Ser. No. 15/653,719, titled “EXTRAMISSIVE SPATIAL IMAGINGDIGITAL EYE GLASS APPARATUSES, METHODS AND SYSTEMS FOR VIRTUAL ORAUGMEDIATED VISION, MANIPULATION, CREATION, OR INTERACTION WITH OBJECTS,MATERIALS, OR OTHER ENTITIES” filed Jul. 19, 2017, now U.S. Pat. No.10,168,791, which is a continuation of U.S. patent application Ser. No.14/147,199 titled “EXTRAMISSIVE SPATIAL IMAGING DIGITAL EYE GLASSAPPARATUSES, METHODS AND SYSTEMS FOR VIRTUAL OR AUGMEDIATED VISION,MANIPULATION, CREATION, OR INTERACTION WITH OBJECTS, MATERIALS, OR OTHERENTITIES” filed Jan. 3, 2014, now U.S. Pat. No. 9,720,505, which claimsthe benefit under 35 U.S.C. § 119(e) of U.S. Provisional PatentApplication Ser. No. 61/748,468 titled “HEAD MOUNTED DISPLAY CONNECTEDWITH COMPUTATIONAL DEVICE AND DEPTH-SENSOR WITH SOFTWARE APPLICATIONSAND USER INTERFACE” filed on 3 Jan. 2013 and U.S. Provisional PatentApplication Ser. No. 61/916,773 titled “VIRTUAL OR AUGMEDIATEDTOPOLOGICAL SCULPTING, MANIPULATION, CREATION, OR INTERACTION WITHDEVICES, OBJECTS, MATERIALS, OR OTHER ENTITIES” filed Dec. 16, 2013, andall of the aforementioned applications are herein expressly incorporatedby reference in their entirety for all purposes.

BACKGROUND

Sensor technology is employed in contexts to monitor and/or measureinformation about an environment, such as the use of temperature sensorsin industrial processes. Mobile devices employ increasingly powerfulprocessing capabilities to support various applications, such astelephony and other mobile communications.

SUMMARY

In accordance with an aspect, there is provided a sensing and displayapparatus, including: an interface assembly including: a first interfacemodule configured to interface with a first computationally mediatedspace (for example, a scene, subject matter, or the like), the firstinterface module configured to exchange sensor signals and effectorsignals with the first space; and a second interface module configuredto interface with a second computationally mediated space (for example,a user's own personal viewing space in or around their digital eye glassspace), the second interface module configured to provide or exchangeeffector signals, and, in some embodiments, also sensor signals (forexample, an eye-tracker, or the like) with the second space, and theeffector signals being user presentable, at least in part, in any one ofthe first space and the second space.

In accordance with an aspect, there is provided a sensing and displayapparatus, including: an interface assembly, including: a firstinterface module configured to interface with sensor signalsrepresenting a sensory phenomenon and/or phenomena, the signals and/orphenomena received, in one implementation, from a first reality space,e.g., a visual-reality space, and in one implementation alsorepresenting sensory phenomena received from a second visual-realityspace; and a second interface module configured to interface witheffector signals representing, displaying, presenting, providing, and/orthe like sensory phenomena to the first visual-reality space and/or tothe second visual-reality space.

In accordance with an aspect, there is provided a method, including: anoperation, including receiving a sensor signal representing sensoryphenomena received from a first space and from a second space.

In accordance with an aspect, there is provided a sensing and displayapparatus, including: an interface assembly configured to interface witha first space and with a second space, and the interface assemblyconfigured to convey sensor signals and effector signals associated withthe first space and the second space; a processing apparatus operativelycoupled to the interface assembly, and the processing apparatusconfigured to process the sensor signals and the effector signalsconveyed by the interface assembly; and a memory assembly configured totangibly embody a processing program, including a sequence of programmedinstructions configured to direct the processing apparatus to executeoperations on the sensor signals and the effector signals.

In accordance with an aspect, there is provided a user interface,including: a first interface section configured to display phenomenaderived from a first space; and a second interface section configured todisplay phenomena derived from a second space.

In accordance with an aspect, there is provided a sensing and displayapparatus, including: a first phenomenon interface configured tooperatively interface with a phenomenon and/or phenomena in a firstspace; and a second phenomenon interface configured to operativelyinterface with a phenomenon and/or phenomena in a second space.

In accordance with an aspect, there are provided other aspects asidentified in the claims.

Other aspects and features of the non-limiting embodiments (examples,options, etc.) may now become apparent upon review of the followingdetailed description of the non-limiting embodiments with theaccompanying drawings.

DETAILED DESCRIPTION OF THE DRAWINGS

The non-limiting embodiments may be more fully appreciated by referenceto the following description of the non-limiting embodiments when takenin conjunction with the accompanying drawings, in which:

FIG. 1A depicts a schematic example of a sensor and display apparatus1300 in one embodiment;

FIG. 1AA depicts a schematic example of a method associated with thesensory and display apparatus 1300 of FIG. 1A in one embodiment;

FIG. 1B depicts a schematic example of a sensory and display apparatus1300 in one embodiment;

FIG. 1C depicts a schematic example of a sensory and display apparatus1300 in one embodiment;

FIG. 1D depicts a schematic example of a sensory and display apparatus1300 in one embodiment;

FIG. 1E depicts a schematic example of a sensory and display apparatus1300 in one embodiment;

FIG. 1EE depicts a signal processing system in one embodiment thataccepts input from a light sensing or light sensing and effectory(luminary) apparatus, as well as from an acoustic sensing or acousticsensing and effectory (e.g., sonar) apparatus;

FIG. 1F depicts examples of interacting with shared objects using anembodiment of the sensory and display apparatus 1300 of FIG. 1E;

FIG. 1G depicts an example of interacting with shared objects along anabakographic trajectory in one embodiment using the sensory and displayapparatus 1300 of FIG. 1E;

FIG. 1H depicts an example of a method for interacting with sharedobjects using the sensor and display apparatus 1300 of FIG. 1E;

FIG. 2A depicts another schematic example of the sensory and displayapparatus 1300 of FIG. 1E;

FIG. 2B depicts Video Orbits stabilization and comparametric alignment,and the like in one embodiment;

FIG. 2C depicts an absement-based signal processor in one embodiment;

FIG. 2D depicts a toposculpting system in one embodiment;

FIG. 2E depicts further details of the toposculpting system in oneembodiment;

FIG. 2F depicts a hand-based toposculpting mesher in one embodiment;

FIG. 2G depicts an embodiment of an inverse surface and a meta table,which in some implementations may be referred to as a METAtableSousface™ system;

FIG. 3 depicts another schematic example of the sensory and displayapparatus 1300 of FIG. 1E in one embodiment;

FIG. 4 depicts another schematic example of the sensory and displayapparatus 1300 of FIG. 1E in one embodiment;

FIG. 5 depicts an example of a diagram indicating timing and sequencingoperations of the sensory and display apparatus 1300 of FIG. 1E in oneembodiment;

FIG. 5A depicts an example of a method of the sensory and displayapparatus 1300 of FIG. 1E in one embodiment;

FIG. 6 depicts an example of a real-time augmediated reality sharedamong multiple participants (users) using the sensory and displayapparatus 1300 of FIG. 1E in one embodiment;

FIG. 7 depicts another example of a real-time augmediated reality sharedamong multiple participants (users) using the sensory and displayapparatus 1300 of FIG. 1E in one embodiment;

FIG. 8A depicts a schematic example of a user interface 800 for use withthe sensory and display apparatus 1300 of FIG. 1E in one embodiment;

FIG. 8B depicts a schematic example of a user interface 800 for use withthe sensory and display apparatus 1300 of FIG. 1E in one embodiment;

FIG. 8C depicts a schematic example of a user interface 800 for use withthe sensory and display apparatus 1300 of FIG. 1E in one embodiment;

FIG. 9A depicts a schematic example of a user interface 800 for use withthe sensory and display apparatus 1300 of FIG. 1E in one embodiment;

FIG. 9B depicts a schematic example of a user interface 800 for use withthe sensory and display apparatus 1300 of FIG. 1E in one embodiment;

FIG. 10A depicts a schematic example of a user interface 800 for usewith the sensory and display apparatus 1300 of FIG. 1E in oneembodiment;

FIG. 10B depicts a schematic example of a user interface 800 for usewith the sensory and display apparatus 1300 of FIG. 1E in oneembodiment;

FIG. 11A depicts a schematic example of a user interface 800 for usewith the sensory and display apparatus 1300 of FIG. 1E in oneembodiment;

FIG. 11B depicts a schematic example of a user interface 800 for usewith the sensory and display apparatus 1300 of FIG. 1E in oneembodiment;

FIG. 11C depicts a schematic example of a user interface 800 for usewith the sensory and display apparatus 1300 of FIG. 1E in oneembodiment;

FIG. 12 depicts a schematic example of a user interface 800 for use withthe sensory and display apparatus 1300 of FIG. 1E in one embodiment;

FIG. 13 depicts a schematic example of a sensory and display apparatus1300 of FIG. 1E in one embodiment; and

FIG. 14 depicts a schematic example of the sensory and display apparatus1300 of FIG. 13 in one embodiment.

The drawings are not necessarily to scale and may be illustrated byphantom lines, diagrammatic representations and fragmentary views. Incertain instances, details not necessary for an understanding of theembodiments (and/or details that render other details difficult toperceive) may have been omitted.

Corresponding reference characters indicate corresponding componentsthroughout the several figures of the drawings. Elements in the severalfigures are illustrated for clarity and have not necessarily been drawnto scale. For example, the dimensions of some of the elements in thefigures may be emphasized relative to other elements for facilitatingunderstanding of the various presently disclosed embodiments.

LISTING OF REFERENCE NUMERALS USED IN THE DRAWINGS

2B subject matter, or true subject matter

100 optical sensor

110 removable shade

120 sensor-interface unit

130 surface

140 public subject matter

141 manifold

142 body part

143 body part

144 trace

145 body part

145 manifold

146 manifold

147 body part

150 network connection

160 computer

170 projection

180AM abakographic manifold, or manifold

180DEG digital eye glass, or glass, or spatial imaging glass

180D display, or display units, or stereo display

180F finger

180G geophone

180H head strap

180IMU inertial sensor

180IRR infrared receiver

180IRT infrared transmitter

180L LiDAR unit

180MD hand

180MS hand

180M mindstrap

1800 object

180P object

180Q object

180T tether clip

180TT thumb

180U user

180VR visible light receiver

180VT visible light transmitter

180V vision system

180 digital eye glass

180U user

181BB object

181DEG glass, or digital eye glass, or spatial imaging glass

181G geophone

181IRR receiver

181L LiDAR unit

181MD hand

181MS hand

181M menu selection

181SM spat menu

181U user

181 display unit, or display units

182 geophone

183 head-mountable assembly

184 inertial measurement unit

185 infrared receiver

186 infrared transmitter

187 LiDAR unit

188 back-of-head band

189 clip

190 digital eye glass

191 visible-light receiver

192 visible-light transmitter

193 vision system

194 geophone

195 infrared receiver

196 digital eye glass

197 LiDAR unit

198 eyes, or naked eyes

199A oculus dexter point-of-eye

199B oculus sinister point-of-eye

200 tool point

201 drawing tool

202 virtual subject matter

203 geophonic sensor

204 inertial measurement unit

240 public subject matter

2D8 lamps

2F8 lamps

330 food items

331 burners

334 cooktop

340 secret subject matter

341 sink

400 taction detector

401 homography intrusion detector

402 confluence sensor

403 video-orbits stabilization program

404 comparametric compositor

405 superposimetric compositor

406 comparametric analysis program

407 superposimetric analyzer

408 superposimetric spatial imaging program

409 comparametric compositor

410 spatial-imaging multiplexer

411 time-division multiplexer

412 collaborative gesture-based interface

413 gesture-tracking detector

414 neural network

415 best-fit optimizer

416 gradient descenter

417 regularizer

418 overflow penalizer

419 human-gesture recognition program

420 bubble-metaphor generator

421 spherical volumetric intrusion estimator

422 volume-intrusion detector

423 bubble-bursting program

424 learning software

430 pattern

500 pulse train signal

502 operation

504 operation

506 operation

508 operation

509 pulse train signal

510 weaker illumination signal

511 weaker illumination signal

513 method

520 medium illumination signal

521 medium illumination signal

530 stronger illumination signal

531 stronger illumination signal

540 weaker illumination signal

541 weaker illumination signal

550 medium illumination signal

551 medium illumination signal

560 stronger illumination signal

561 stronger illumination signal

570 first time slot

571 second time slot

572 third time slot

573 fourth time slot

574 fifth time slot

575 sixth time slot

580 method

582 operation

584 operation

586 operation

588 operation

600 object

601 surfaces

602 recognizable surface texture

603 image icon

610 gesture input

611 gesture input

612 gesture input

630 wireless communications module

631 wireless communications module

632 wireless communications module

700 real object

710 hand

720 bursting bubble

800 user interface

801 user

802 view

803 bubble set

804 work bubble

805 media bubble

806 play bubble

807 social bubble

808 settings bubble

810 setting type

811 setting type

812 setting type

813 setting type

8×8 eight

902 interface assembly

903 first interface module

904 sensor signal, or sensor signals

905 second interface module

906 effector signal, or effector signals

907 processing program, or program

908 processing apparatus

909 memory assembly, or non-transitory machine-readable storage medium

910 first sensory-phenomenon sensor, or sensory-phenomenon sensor

911 electronic network connection

912 first sensory-phenomenon effector, or sensory-phenomenon effector

914 second sensor-phenomenon effector, or sensory-phenomenon effector

916 second sensory-phenomenon sensor, or sensory-phenomenon sensor

1000 first augmediated-reality space

1002 second augmediated-reality space

1100 method

1102 operation

1104 operation

1106 operation

1108 operation

1110 operation

1300 apparatus, or display apparatus

1302 first phenomenon interface

1304 processing assembly

1306 second phenomenon interface

1308 depth map

1C10 topological sensor

1C20 maniplay

1C30 sustainment detector

1C40 maniplay terminator

1050 gesture sensor

1060 object deleter

1C70 spat menu

1C80 submanifold mover

1C90 manifold mover

2B00 subject matter

2B05 subject matter

2B06 subject matter

2B10 subject matter

2B11 corner

2B12 corner

2B13 corner

2B14 corner

2B15 subject matter

2B16 subject matter

2B20 subject matter

2B25 subject matter

2B26 subject matter

2B30 subject matter

2B35 subject matter

2B36 subject matter

2C10 process

2C15 process variable

2C20 adder

2C25 error signal

2C30 kinematic processors

2C35 processor

2C40 processed kinematics signals

2C45 adder

2C50 signal

2C55 quantities

2C60 quantities

2C70 signal

2D21 rays

2D22 rays

2D23 visible rays

2D24 rays

2D32 ray

2D34 ray

2D40 exposure

2D51 character exposure

2D55 character exposure

2D60 character exposure

2D70 tabletop surface

2D90 exposure

2D91 abakograph

2D92 abakograph

2E10 abakograph

2E11 weak exposure

2E12 exposure

2E13 exposure

2F64 lamps

1EE10 acoustic trusion input

1EE20 visual trusion input

1EE25 acoustic trusion sensor

1EE30 visual trusion sensor

1EE35 multidimensional trusion signal

1EE40 node

1EE45 connection

1EE50 node

1EE60 connection

1EE70 node

1EE80 taction signal

1EE85 taction signal

1EE90 taction signal

1EE91 intrusion detector

1EE92 touch detector

1EE93 extrusion detector

1EE94 signal

1EE95 signal

1EE96 signal

1EE97 gesture sensor

1EE98 various output signals

2G110 metable label

2G120 object

2G130 metable circular object

2G140 objects

2G150 objects

2G160 toposculpting wand

2G161 ring

2G162 trigger

2G163 detacheable grip

2G164 handle

2G165 shaft

2G168 abakograph

2G169 additional apparatus, or apparatus

2G181 manifold

2G182 manifold

2G184 manifold

2G189 manifold

2G190 gesture band

2G193 object

2G199 middle finger

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the described embodiments (examples, options,etc.) or the application and uses of the described embodiments. As usedherein, the word “exemplary” or “illustrative” means “serving as anexample, instance, or illustration.” Any implementation described hereinas “exemplary” or “illustrative” is not necessarily to be construed aspreferred or advantageous over other implementations. All of theimplementations described below are exemplary implementations providedto enable making or using the embodiments of the disclosure and are notintended to limit the scope of the disclosure. For purposes of thedescription herein, the terms “upper,” “lower,” “left,” “rear,” “right,”“front,” “vertical,” “horizontal,” and derivatives thereof shall relateto the examples as oriented in the drawings. Furthermore, there is nointention to be bound by any expressed or implied theory presented inthe preceding technical field, background, brief summary or thefollowing detailed description. It is also to be understood that thespecific devices and processes illustrated in the attached drawings, anddescribed in the following specification, are exemplary embodiments(examples), aspects and/or concepts. Hence, specific dimensions andother physical characteristics relating to the embodiments disclosedherein are not to be considered as limiting, except in the context ofany claims which expressly states otherwise. It is understood that “atleast one” is equivalent to “a”. The aspects (examples, alterations,modifications, options, variations, embodiments and any equivalentthereof) are described with reference to the drawings; it should beunderstood that the descriptions herein show by way of illustrationvarious embodiments in which claimed inventions may be practiced and arenot exhaustive or exclusive. They are presented only to assist inunderstanding and teach the claimed principles. It should be understoodthat they are not representative of all claimed inventions. As such,certain aspects of the disclosure have not been discussed herein. Thatalternate embodiments may not have been presented for a specific portionof the invention or that further undescribed alternate embodiments maybe available for a portion is not to be considered a disclaimer of thosealternate embodiments. It will be appreciated that many of thoseundescribed embodiments incorporate the same principles of the inventionand others are equivalent. Thus, it is to be understood that otherembodiments may be utilized, and functional, logical, organizational,structural and/or topological modifications may be made withoutdeparting from the scope and/or spirit of the disclosure.

FIG. 1A depicts a schematic example of a sensing and display apparatus1300 in one embodiment.

In accordance with FIG. 1A, the sensing and display apparatus 1300includes (and is not limited to) an interface assembly 902. In oneimplementation, the interface assembly 902 is configured to interfacewith sensor signals 904 representing sensory phenomena received fromand/or detected in a first augmediated-reality space 1000. The firstaugmediated-reality space 1000 may be called an augmented reality spaceand/or a mediated-reality space). The sensor signals 904 may also, inone implementation, represent sensory phenomena received from and/ordetected in one or more additional spaces, such as a secondaugmediated-reality space 1002. In one implementation, the firstaugmediated-reality space 1000 and the second augmediated-reality space1002 are exclusive (e.g., do not overlap each other), while in anotherimplementation, all or part of the first augmediated-reality space 1000and the second augmediated-reality space 1002 may be overlapping,coincident, coextensive, and/or the like. In some implementations, thefirst augmediated-reality space 1000 and the second augmediated-realityspace 1002 may be alternately exclusive and/or coextensive depending onvarious factors, such as the identity of users, the identity of one ormore software applications running on the device, the character and/orvalue of detected and/or measured ambient factors (e.g., light levels,time, GPS, and/or the like), and/or the like. The interface assembly 902may also be configured, in one implementation, to interface witheffector signals 906 representing, effecting, displaying, providing,and/or the like sensory phenomena to the first augmediated-reality space1000 and to the second augmediated-reality space 1002. As used herein invarious implementations, to interface may include to exchange, convey,receive, and/or the like.

In accordance with another example, the sensing and display apparatus1300 includes (and is not limited to) the interface assembly 902including a first interface module. In one implementation, the firstinterface module may be configured to interface (or to be orientedtoward or face) a first augmediated-reality space 1000. In anotherimplementation, the first interface module may be configured to interactin other ways with the first augmediated-reality space 1000. Forexample, in audio embodiments, the first interface module may includeone or more microphones that may, in some implementations, be orientabletowards sound sources. The first interface module may be configured toconvey and/or exchange the sensor signals 904 and effector signals 906with the first augmediated-reality space 1000. The interface assembly902 may also include a second interface module in one implementation.The second interface module may be configured to interface face (or beoriented toward or face) a second augmediated-reality space 1002. Thesecond interface module may be configured to convey and/or exchange thesensor signals 904 and effector signals 906 with the secondaugmediated-reality space 1002. In one implementation, the effectorsignals 906 (and/or representations of the effector signals 906) may beuser presentable, at least in part, in one or both of the firstaugmediated-reality space 1000 and the second augmediated-reality space1002.

The sensor signals 904 and the effector signals 906 are presentable invarious forms, at least in part, for user-sensory presentation and/orconsumption, such as by way of visual display, audio presentation, usermanipulation, and/or any other form of user consumption in one or bothof the first augmediated-reality space 1000 and the secondaugmediated-reality space 1002.

In accordance with an option, the sensing and display apparatus 1300 mayalso include a processing apparatus 908 configured to operatively coupleto the interface assembly 902. In one implementation, the processingapparatus 908 may be configured to process the sensor signals 904interfaced with and/or otherwise in association with the interfaceassembly 902, and/or to process the effector signals 906 interfaced withand/or in association with the interface assembly 902. The processingapparatus 908 may include processing (e.g., augmenting, modifying,and/or the like) the sensor signals 904 and/or the effector signals 906.

In accordance with an option, the sensing and display apparatus 1300 mayinclude an interface assembly 902. In one implementation, the interfaceassembly 902 may include a first interface module 903 configured tointerface with sensor signals 904, such as representing sensoryphenomena received from a first augmediated-reality space 1000, and/oralso representing sensory phenomena received from the secondaugmediated-reality space 1002. In one implementation, the interfaceassembly 902 may also include a second interface module 905 configuredto interface with effector signals 906 representing, displaying,providing, projecting, and/or the like sensory phenomena to the firstaugmediated-reality space 1000 and to the second augmediated-realityspace 1002. Other configurations of the interface assembly 902 arepossible in view of the description provided herein.

For example, the first augmediated-reality space 1000 is an augmented ormediated-reality environment (space) configured to be a space that isshared among two or more users (a publicly shared space). The secondaugmediated-reality space 1002 is an augmented or mediated-realityenvironment (space) configured to be a space used by a user (e.g., apersonal space), or a space that may be shared (e.g., for use) by alimited number of users (e.g., a private space), or a space that may beshared by all users of the first augmediated-reality space 1000.

For example, the second augmediated-reality space 1002 is configured toprovide (to the user or the users) a live (direct or indirect) view ofthe first augmediated-reality space 1000. In one implementation, thefirst augmediated-reality space 1000 includes a physical (real world)environment having physical elements that may be augmented, supplementedand/or mediated, (e.g., augmediated) by computer-generated sensoryphenomena (such as sound, video, graphics, tactile elements, and/or thelike), such as may be projected onto the physical elements contained inthe first augmediated-reality space 1000, or sensory phenomena that aremodified (e.g., by computer) with the modified version presented to theuser (for example, diminished reality to help the user sense andunderstand the world better by removal of “clutter”, such a diminishedreality being another example of an augmediated reality). The apparatus1300 thus may be configurable, in some implementations, to incorporateany combination of physical elements and/or information, virtualelements and/or information, and/or elements that are effected,projected, subtracted, and/or the like onto or from the environment,into displayable subject matter for presentation via an output display,e.g., to a user of the device. In some implementations, effector signalsand/or other computer-generated sensory phenomena may be employed forany of a variety of uses beyond and/or in addition to illumination,augmentation and/or demarcation of subject matter in the environment.For example, in one implementation, effector signals and/or othercomputer-generated sensory phenomena may include red-eye reduction flashcapabilities, wherein a flash is preceded by a series of low-powerflashes to trigger contraction of the pupil. In another example, in oneimplementation, effector signals and/or other computer-generated sensoryphenomena may include a flash, strobe, and/or the like for alerting,warning, blinding, disabling, disorienting, and/or the like.

In one implementation, the first augmediated-reality space 1000 may be ashared space, shared by multiple users. In one implementation, userssharing a first augmediated-reality space 1000 may be located near eachother, for example, e connecting via any type of network connection,such as (and not limited to) a local area network, the Bluetooth™network, the WiFi™ network, and/or the like, while in anotherimplementation, users may be located remotely from each other, forexample, and are connected via a wide area network, cellular network.TCP/IP network, and/or the like. The second augmediated-reality space1002 may be a shared space as well and may include options for limitedsharing with other users. The second augmediated-reality space 1002 may,in one implementation, belongs to and/or be assigned to a predeterminedand/or selected user drawn from a group of users. In this sense, thesecond augmediated-reality space 1002 is a personal space of apredetermined user. The user of the second augmediated-reality space1002 may permit access to the second augmediated-reality space 1002 tono users, to a subset of the users or all of the users associated withthe first augmediated-reality space 1000 (e.g., in accordance with thedesire or control of the user assigned to the second augmediated-realityspace 1002).

In various implementations, one or both of the first augmediated-realityspace 1000 and one or more instances of the second augmediated-realityspace 1002 may be network accessible, such as via any type of network,such as (and not limited to) a local area network, a wide area network,a cellular network, a WiFi™ network, a Bluetooth™ network, a TCP/IPenabled network, and/or the like. The Internet protocol suite is thenetworking model and a set of communications protocols used for theInternet and similar networks. The Internet protocol suite is commonlyknown as TCP/IP, because its most important protocols, the TransmissionControl Protocol (TCP) and the Internet Protocol (IP), were the firstnetworking protocols defined in this standard.

For example, for the case where there are seven users associated withthe first augmediated-reality space 1000, each member of that group ofusers has access to the first augmediated-reality space 1000, and inthis way, the first augmediated-reality space 1000 is treated as acommunal space (e.g., a public space). The phenomena (sensory phenomena)associated with the first augmediated-reality space 1000 may beexperienced or shared, in a communal way, by the users who access thefirst augmediated-reality space 1000 by way of each user's respectiveinstance of the sensory and display apparatus 1300.

In one implementation, each user may be assigned their own instance of asecond augmediated-reality space 1002 via their respective instance ofthe sensory and display apparatus 1300. The apparatus 1300 is configuredto permit the user of the apparatus 1300 access to the firstaugmediated-reality space 1000, and access to their instance of thesecond augmediated-reality space 1002. The apparatus 1300 is configuredto allow the user to permit a degree of access (e.g., no access, limitedaccess, full access) to their instance of the second augmediated-realityspace 1002 to the other users of the first augmediated-reality space1000. In one implementation, ranges of access may be applied and/orenforced, such as access for set and/or limited times and/or locations;profile, permissions and/or role-based access; task and/oraccomplishment-based access; and/or the like. Such ranges of access maybe enforced, in one implementation, based on access records linked todata records associated with access triggers and/or parameter values. Inan implementation of shared access, a first user's instances of thesecond augmediated-reality space 1002, shared to the firstaugmediated-reality space 1000 of one or more other users, may beannotated, tagged, and/or otherwise labeled. For example, a secondaugmediated-reality space 1002 instance may include one or morecharacteristic annotations, logos, watermarks, drawings, graphics,photos, names, labels, tags, colors, shapes, sounds, and/or the like,such as may identify the first user as associated with the secondaugmediated-reality space 1002. In one implementation, a user withaccess privileges to two or more instances of the secondaugmediated-reality space 1002 may select viewing options and/or directviews of the plurality of the second augmediated-reality space 1002,such as by opening, closing, moving, resizing, maximizing, minimizing,orienting, squeezing, stretching, sculpting, and/or the like selectedshared second space views. In one implementation, a user with accessprivileges to at least one shared second space may be presentedconflicting and/or inconsistent views of the same object and/or physicalspace between the first augmediated-reality space 1000, the secondaugmediated-reality space 1002 and, in some instances, one or moreadditional shared second spaces. For example, in a game scenario, avirtual object may appear to some users but not others based on gameparameters. A user viewing two shared second spaces, one with thevirtual object and one without, may thus perceive different views of thesame physical space (e.g., one with the virtual object and one without).In one implementation, views of the second augmediated-reality space1002 may be overlaid with each other and/or with the views of the firstaugmediated-reality space 1000. In one implementation, subsets ofobjects, virtual objects, and/or other display components may bepresented within particular shared views of a given secondaugmediated-reality space 1002, such as based on the roles, privileges,settings, and/or the like of the receiving user.

For instance, a first user, by way of a configuration setting providedby the sensory and display apparatus 1300, may be able to restrictaccess to their instance of the second augmediated-reality space 1002 sothat no other users of the first augmediated-reality space 1000(accessed by way of respective instances of the sensing and displayapparatus 1300) may have access or shared access to the secondaugmediated-reality space 1002 of the first user. The first user hasdecided to make his instance of the second augmediated-reality space1002 (via the sensing and display apparatus 1300) private andinaccessible to the other users having their own instance of theapparatus 1300.

In an example, a second user of the first augmediated-reality space1000, by way of the configuration setting in their instance of theapparatus 1300, permits a third user and a fourth user to access thesecond augmediated-reality space 1002 assigned to the second user. Thethird user and the fourth user may decide whether to share their ownrespective instances of the second augmediated-reality space 1002 withthe second user.

The remaining users (associated with the first augmediated-reality space1000) may set up access of their instance of the secondaugmediated-reality space 1002 (via their respective instances of thesensing and display apparatus 1300) in accordance with their respectiveneeds, choices, roles, privileges, and/or the like, as indicated above(for example).

For instance, in one embodiment, an electronic game (such as anaugmented or mediated reality game) may be set up, in which two opposingteams of users match up their skills in a team effort. Therefore, afirst team of users of the first augmented or first augmediated-realityspace 1000 may configure their instance of the sensing and displayapparatus 1300 in such a way that each member of the team has access toeach respective instance of the second augmediated-reality space 1002 ofeach member of the first team. This way, the team members may chat(exchange user communications) amongst themselves while they play thegame and strategize for victory over the second team of users while thegame is played out in the first augmediated-reality space 1000. In thefirst augmediated-reality space 1000, all users have access to the firstaugmediated-reality space 1000 and may interact while the game is playedout. The electronic game may be a war game, a chess game, and any othersort of game that engages team players, etc. The size of a team may beat least one user. The users may, in one example, be physically locatedin remote locations.

In accordance with an option, multiple users, each wearing an instanceof the sensing and display apparatus 1300, may experience a sharedcomputer-mediated reality (e.g., in the first augmediated-reality space1000 and/or in the second augmediated-reality space 1002). In anotheroption, users not wearing the sensing and display apparatus 1300 mayeven share some elements of the computer-mediated reality in the firstaugmediated-reality space 1000 and may also participate within the firstaugmediated-reality space 1000.

More than one user may wear their instance of the sensing and displayapparatus 1300 while viewing the same subject matter. In the case ofmultiple instances of the apparatus 1300, each instance of the sensingand display apparatus 1300 may, in one implementation, be configured toperform spatial-imaging functions (operations) while sharing data(phenomenon signals, for example) with one or more other instances ofthe sensing and display apparatus 1300 in the first augmediated-realityspace 1000 and/or the second augmediated-reality space 1002.

The sensing and display apparatus 1300 may be configured to provide anaugmediated reality environment (e.g., augmediated reality, a mediatedreality experience) via the first augmediated-reality space 1000 and/orvia the second augmediated-reality space 1002, which may be controllablethrough the movements of the appendages of the user (hands, fingers,arms, legs, feet, etc.) interacting either in free-space, with one ormore virtual objects, and/or interacting with a tangible (physical)object, such as a flat surface (e.g., a table, a countertop, a cooktop,the floor, the ground, and/or the like).

In one implementation, the first augmediated-reality space 1000 and thesecond augmediated-reality space 1002 may be configured to provideimagery, such as three-dimensional imagery, to the user or users. Thefirst augmediated-reality space 1000 and the second augmediated-realityspace 1002 may, in some implementations, be referred to as a cyborgspace and/or a mediated reality environment. The firstaugmediated-reality space 1000 and the second augmediated-reality space1002 may be configured to provide access (e.g., a view) to a realitythat may be modified (e.g., diminished, augmediated, and/or the like) bythe processing apparatus 908. The processing apparatus 908 may, in oneimplementation, be configured to enhance perception of reality for theuser and/or users. In one implementation, the processing apparatus 908is configured to provide and/or facilitate the first augmediated-realityspace 1000 and the second augmediated-reality space 1002 in real-timeand in a semantic context with environmental elements, such as sportsscores of sporting events during a match. For example, by adding and/orusing computer-vision devices and/or object-recognition devices, theinformation about the surrounding real world of the user or usersbecomes interactive and may be manipulated digitally by the processingapparatus 908 of the sensing and display apparatus 1300. In oneimplementation, meta (e.g., artificial, virtual, additional,augmediating, and/or the like) information about the firstaugmediated-reality space 1000 and the second augmediated-reality space1002 may be overlaid on the physical elements associated with (e.g.,located in) the first augmediated-reality space 1000. By way of example,the first augmediated-reality space 1000 may be configured to beaccessible by users (e.g., more than two users), and the secondaugmediated-reality space 1002 may be configured to be accessible by asingle user or by any number of users as desired by the user of theirinstance of the second augmediated-reality space 1002. Additionalexamples of the first augmediated-reality space 1000 and the secondaugmediated-reality space 1002 are described below in connection withthe figures. In accordance with an option, the first augmediated-realityspace 1000 includes a spatial augmediated reality environment configuredto augment real world objects and scenes by using a digital projector(an example of an effector assembly) configured to display graphicalinformation onto physical objects located in the firstaugmediated-reality space 1000. The spatial augmediated realityenvironment may be configured to accommodate access to a group of users,thus allowing for collocated collection and/or collaboration betweenusers of the spatial augmediated reality environment.

The sensor signal 904 is generated and/or provided by a sensor and/orsensor assembly in response to receiving, at least in part, one or moresensory phenomena (such as, visual phenomena, auditory phenomena,tactile phenomena, and/or the like). The sensor assembly is configuredto be sensitive to the receipt of, at least in part, the sensoryphenomena (also called sensory stimuli). The sensor assembly isconfigured to detect a physical quality and/or quantity (e.g.,associated with one or more sensory phenomena), and to convert and/ormanifest the physical quality and/or quantity into the sensor signal904. The sensor signal 904 is configured to be readable by an electronicapparatus (such as, the processing apparatus 908 of FIG. 1A, etc.). Thesensor signal 904 may, for example, be an electrical signal, an opticalsignal, or any type of signal embodied in any suitable medium. Anexample of the sensor assembly includes a thermometer, or athermocouple. In another example, the sensor assembly may include atactile-based sensor. Other examples of the sensor assembly areidentified below in connection with FIG. 1E (and other figures).

The effector signal 906 is a signal provided by (transmitted by) aneffector and/or effector assembly. The effector assembly is configuredto produce a desired (predetermined) change in an object (such as animage) in response to receiving an input (e.g., input signal, sensoryphenomena). An example of an effector assembly includes anoptical-projection system. The object may be an optical object, imageobject, video object, and/or the like, and may include an image based onvisible light. In one implementation, the effector assembly may includean actuator configured to actuate or control. In one implementation, anactuator is the mechanism by which a control system acts upon anenvironment. Further examples of the effector assembly are identifiedbelow in connection with FIG. 1E (and other figures).

Referring to FIG. 1B, in accordance with an option, the sensing anddisplay apparatus 1300 may be configured to include the firstsensory-phenomenon effector 912 and the second sensory-phenomenoneffector 914, which may, in some instances, be referred to asspatial-imaging devices. Examples of the first sensory-phenomenoneffector 912 and the second sensory-phenomenon effector 914 include (andare not limited to): an image or video camera and/or projector, aholographic imaging device, a three-dimensional imaging device, alaser-imaging device, a LiDAR device, a time-of-flight imaging device, aRaDAR device, a SoNAR device, a depth camera, a depth sensor, a visiblelight-based device, an infrared-based device, a microwave-based device,a sound-based device, a holography device, a stereoscopy device, athree-dimensional imaging device (in any form), a depth-sensing device,a vision-based device, a shape-from-X device, and/or the like. LiDARstands for Light Detection And Ranging. RaDAR stands for Radio DetectionAnd Ranging. SoNAR stands for Sound Navigation And Ranging.

In one implementation, the interface assembly 902 is configured toprovide a point of interaction and/ or communication between theprocessing apparatus 908 (such as, a computer) and any other entity,such as a printer or a human operator (user).

The processing apparatus 908 is configured to receive and/or read one ormore inputs (e.g., data and/or information), and is also configured toproduce (e.g., provide, write) one or more outputs based on the definedinput received. The processing apparatus 908 is configured to interpretthe defined inputs and the defined outputs as data, facts, information,and/or the like. By way of example (and not limited thereto), theprocessing apparatus 908 may include a combination of: a conversionmodule configured to convert data to another format; a validation moduleconfigured to ensure that supplied data is clean, correct and useful; asorting module configured to arrange items in some sequence and/or indifferent sets; a summarization module configured to reduce detail datato its main points; an aggregation module configured to combine multiplepieces of data; an analysis module configured to collect, organize,analyze, interpret and present data and/or information; and a reportingmodule configured to display or list detail or summary data or computedinformation. There are many assemblies and/or components of theprocessing apparatus 908, that may be utilized to implement theprocessing apparatus 908 of FIG. 1A and/or FIG. 1B.

Referring back to FIG. 1A, the processing apparatus 908 may include, forexample, a central processing unit, a processing unit, a microprocessor,a microcomputer, an array processor, and/or a vector processor (and anyequivalent thereof). The processing apparatus 908 may include a hardwarecircuit within a computer that carries out the instructions of acomputer program by performing arithmetical, logical, and/orinput/output operations. The processing apparatus 908 may include one ormore instances of a processing unit (this case is calledmultiprocessing). The array processor and/or the vector processorincludes multiple parallel computing elements, with no one processingunit considered the center. In the distributed computing model,operations are executed by a distributed interconnected set ofprocessors. In one implementation, the processing apparatus 908 mayinclude a set of dedicated circuits that replace any instructionsoperated or executed by the processing apparatus 908; such dedicatedcircuits may include a programmable logic array (PLA) (and anyequivalent thereof) configured to implement combinational logiccircuits. The PLA has a set of programmable AND gate planes, which linkto a set of programmable OR gate planes, which can then be conditionallycomplemented to produce an output.

The processing apparatus 908 may include a non-transitorymachine-readable storage medium 909, hereafter referred to as the memoryassembly 909. The memory assembly 909 is configured to store data andexecutable programs (programmed instructions) in a format readable bythe processing apparatus 908. Examples of the memory assembly 909 mayinclude computer readable and/or computer writable media, magnetic mediasuch as magnetic disks, cards, tapes, and drums, punched cards and papertapes, optical disks, barcodes and magnetic ink characters, and/or thelike. Examples of machine-readable technologies include magneticrecording, processing waveforms, barcodes, and/or the like. In oneimplementation, optical character recognition (OCR) can be used to allowthe processing apparatus 908 to read information, such as informationthat is readable by humans. Any information retrievable by any form ofenergy can be machine-readable.

In one implementation, the processing apparatus 908 may be configured tointerface with an electronic network connection 911. The electronicnetwork connection 911 (or network access), such as an Internetconnection (or access), may be configured to connect instances of theprocessing apparatus 908 (such as, computer terminals, computers, mobiledevices, cell phones, computer networks, and/or the like) to one or moreelectronic networks thus allowing users to access network services (forexample, e-mail, the World Wide Web, etc.). In some implementations, theprocessing apparatus 908 may be implemented as a remote processingapparatus or a distributed processing apparatus, and may, in whole orpart, be communicatively coupled to one or more instances of theapparatus 1300 via one or more wireless and/or wired communicationnetworks.

In some implementations, the processing apparatus 908 may be configuredto include a user-input assembly (such as, a mouse device, a keyboarddevice, a camera, a touch sensitive display screen, microphone, retinareader, accelerometer, ambient light sensor, GPS, antenna, and/or thelike). The processing apparatus 908 may further be configured to includea user-output assembly (such as, a computer terminal, a televisionscreen or other video-display interface, a projector, a touch sensitivescreen, and/or the like).

The memory assembly 909 is configured to tangibly embody a processingprogram 907, hereafter referred to as the program 907. The program 907includes a sequence of programmed instructions configured to direct theprocessing apparatus 908 to perform (execute) specified operations(tasks, such as reading, writing, processing). The processing apparatus908 executes the program 907, such as by using a processor assembly. Theprogram 907 has an executable form that the processing apparatus 908 mayuse to execute the programmed instructions provided by the program 907.The program 907 may be used in its human-readable source code form, fromwhich executable programs are derived (e.g., compiled), to configureprogrammed instructions to be used or included in the program 907. Theprogram 907 is a collection of program instructions and/or related dataand may be referred to as the software (code). The program 907 directsthe processing apparatus 908 to perform, for example, image-processingoperations, such as of the type described in a wearable computingtextbook authored by S. Mann, entitled “Intelligent Image Processing”,published by John Wiley and Sons, through Wiley Interscience, and IEEEPress, 2001 (incorporated herein by reference).

The memory assembly 909 may be configured to tangibly manifest a userinterface 800, of which an example is depicted in FIG. 8A. The userinterface 800 may, in one implementation, be displayed or provided tothe user of the sensing and display apparatus 1300 (e.g., via the secondaugmediated-reality space 1002). If the user of the secondaugmediated-reality space 1002 desires to set up the configuration ofthe sensing and display apparatus 1300 to share their instance and/orinstances of the second augmediated-reality space 1002 with other users,the configuration of the sensing and display apparatus 1300 may allowthe user to permit access (e.g., to the other users) to the userinterface 800. The user interface 800 may, in one implementation,include a layout of control elements (e.g., graphic-control elementsand/or textual-control elements) in conjunction with the way the program907 of FIG. 1A used by the processing apparatus 908 of FIG. 1A respondsto user activities. In one implementation, the user interface 800 is acomponent facilitating interaction between the user of the sensing anddisplay apparatus 1300 and the sensing and display apparatus 1300itself. The user interface 800 may be used, in some implementations, foreffective operation and control of the processing apparatus 908 on theuser's end, and feedback from the processing apparatus 908 may assistthe user of the processing apparatus 908 in making operational decisions(e.g., decisions on how to operate the processing apparatus 908). Theuser interface 800 may include hardware components (physical components)and/or software components (logical components or virtual components).The user interface 800 is configured, in one implementation, to provide:a user input (e.g., a field) configured to allow the user to control ormanipulate the processing apparatus 908, and an output (e.g., a field)configured to allow the processing apparatus 908 to display and/orindicate, to the user, the effects of the user manipulation via the userinput. The user interface 800 may, in some implementations, also bereferred to as a graphical user interface and/or a human-machineinterface. Other terms for the user interface 800 may includehuman-computer interface (HCI) and man-machine interface (MMI). The userinterface 800 is configured to be displayed for viewing by a user viathe sensing and display apparatus 1300. In one implementation, the userinterface 800 may be presented to the user via a secondsensory-phenomenon effector 914 (e.g., such as depicted in FIG. 1B). Theuser interface 800 may be configured to provide or display to the userone or more fields. In one implementation, a field is a space allocatedfor a particular item of information, such as a user name. Fields may beconfigured to have certain attributes associated with them. For example,some fields may be numeric whereas others may be textual. In someimplementations, every field may have a name, called the field name. Inone implementation, a collection of fields may be referred to as arecord. The user may interact with the fields provided by the userinterface 800 via interfaced circuits or elements.

In accordance with an option, the sensing and display apparatus 1300 mayinclude (and is not limited to) a combination of an interface assembly902, a processing apparatus 908, and a memory assembly 909. Theinterface assembly 902 may be configured, in one implementation, tointerface with a first augmediated-reality space 1000 and with a secondaugmediated-reality space 1002. The interface assembly 902 may beconfigured, in one implementation, to convey sensor signals 904 andeffector signals 906 associated with the first augmediated-reality space1000 and the second augmediated-reality space 1002. The processingapparatus 908 may be configured, in one implementation, to operativelycouple to the interface assembly 902. The processing apparatus 908 maybe further configured to process the sensor signals 904 and the effectorsignals 906 conveyed by the interface assembly 902. The memory assembly909 may be configured, in one implementation, to tangibly embody aprocessing program 907 including a sequence of programmed instructionsconfigured to direct the processing apparatus 908 to execute operationson the sensor signals 904 and/or the effector signals 906.

FIG. 1AA depicts a schematic example of a method 1100 associated withthe sensing and display apparatus 1300 of FIG. 1A.

The method 1100 may be implemented as programmed instructions to beincluded in the program 907 of FIG. 1A; the method 1100 includes theexamples of executable operations (programmed instructions) to beexecuted by the processing apparatus 908 via the program 907.

The method 1100 includes (and is not limited to) an operation 1102,including receiving (reading) the interfaced instances of the sensorsignals 904 representing sensory phenomena received from a firstaugmediated-reality space 1000, and receiving (reading) the interfacedinstances of the sensor signals 904 representing sensory phenomenareceived from a second augmediated-reality space 1002. The interfacedinstances of the sensor signals 904 are received by the processingapparatus 908, such as from the interface assembly 902. Operationalcontrol is passed on to operation 1104.

The method 1100 further includes (and is not limited to) an operation1104, including providing (transmitting, writing) the interfacedinstances of the effector signals 906 representing sensory phenomena tothe first augmediated-reality space 1000, and providing (e.g.,transmitting, writing) interfaced instances of the effector signals 906to the second augmediated-reality space 1002. The interfaced instancesof the effector signals 906 are provided by the processing apparatus 908to the interface assembly 902. Operational control is passed on tooperation 1106.

The method 1100 further includes (and is not limited to) an operation1106, including processing the sensor signals 904 (received from theinterface assembly 902), and further including processing the effectorsignals 906 (received from the interface assembly 902). Processing ofsensor signals may include, but is not limited to, modifying,augmenting, supplementing, complementing, enhancing, diminishing,obscuring, blocking, and/or the like sensor signal data. Operationalcontrol is passed on to other operations associated with the processingapparatus 908.

In some implementations, the method 1100 may include additionaloperations to be executed by the processing apparatus 908. Theprogrammed instructions for the program 907 are derived from the method1100.

For example, the method 1100 further includes (and is not limited to) anoperation 1108, including routing (e.g., switching, mixing, and/or thelike) the sensor signals 904 between the first augmediated-reality space1000 and the second augmediated-reality space 1002. Operational controlis passed on to operation 1110. For example, switching may occur betweentwo sensors of different accuracy and/or precision; between sensorslocated in different environments, between a given sensor at differenttimes (e.g., based on measured values stored in a sensor record);sensors sensitive to different and/or complementary sensory phenomena(e.g., color and intensity; pitch and volume; thermal and visible;and/or the like); sensors associated with different users and orapparatus 1300 instances; and/or the like.

For example, the method 1100 further includes (and is not limited to) anoperation 1110, including routing (e.g., switching, mixing and/or thelike) the effector signals between the first augmediated-reality space1000 and the second augmediated-reality space 1002. Operational controlis passed on to other operations of the processing apparatus 908. Forexample, switching may occur between two effectors in differentlocations; having different and/or complementary outputs, products,and/or the like (e.g., visible-light versus infrared light; illuminationvs. sound); associated with different users and or apparatus 1300instances; and/or the like.

It will be appreciated that additional operations of the processingapparatus 908 may be provided for a multitude of combinations and/orpermutations (of the programmed instructions), such as depending on thenumber of users involved in the first augmediated-reality space 1000,and/or depending on which users share their instance of the secondaugmediated-reality space 1002 with selected other users of the firstaugmediated-reality space 1000 of FIG. 1A, and in association with theoperations described in association with the figures.

It will be appreciated that the order of operations in the method 1100depicted in FIG. 1AA do not have to be executed in a sequential manner,and any execution order of the operations of the method 1100 and/or ofthe program 907 of FIG. 1A may be used via programmed computerinstructions. In some implementations, all or any subset of theaforementioned operations, and/or the like, may be performed inparallel, partial sequence, series, and/or any combination thereof inaccordance with a given embodiment, implementation and/or application.

FIG. 1AA also depicts a data flow diagram in one embodiment, thatdepicts the processing apparatus 908 configured to route (e.g., switch,mix, and/or the like) the sensor signals 904 between the firstaugmediated-reality space 1000 and the second augmediated-reality space1002. The processing apparatus 908 is configured to route (e.g., switch,mix, and/or the like) the effector signals between the firstaugmediated-reality space 1000 and the second augmediated-reality space1002. In addition, the processing apparatus 908 may be configured toroute the sensor signals 904 received from the first augmediated-realityspace 1000 (such as via a first sensory-phenomenon sensor 910 depictedin FIG. 1B) to the second augmediated-reality space 1002 (such as viathe second sensory-phenomenon effector 914 depicted in FIG. 1B). Inaccordance with an option, the processing apparatus 908 is configured toroute the sensor signals 904 received from the secondaugmediated-reality space 1002 (such as via a second sensory-phenomenonsensor 916 depicted in FIG. 1B) to the first augmediated-reality space1000 (such as via a first sensory-phenomenon effector 912 depicted inFIG. 1B). It will be appreciated that other options are possible forrouting the sensor signals and/or the effector signals.

FIG. 1AA also depicts a variety of programs (e.g., programmedinstructions), data tables, and/or the like for use, in someembodiments, with the sensing and display apparatus 1300 depicted in thefigures. The programs may include, but are not limited to: a tactiondetector 400, a homography intrusion detector 401, confluence sensor402, a video-orbits stabilization program 403, a comparametriccompositor 404, a superposimetric compositor 405, a comparametricanalysis program 406, a superposimetric analyzer 407, a superposimetricspatial imaging program 408, a comparametric compositor 409, aspatial-imaging multiplexer 410, a time-division multiplexer 411, acollaborative gesture-based interface 412, a gesture-tracking detector413, a neural network 414, a best-fit optimizer 415, a gradientdescenter 416, a regularizer 417, an overflow penalizer 418, ahuman-gesture recognition program 419, a bubble-metaphor generator 420,a spherical volumetric intrusion estimator 421, a volume-intrusiondetector 422, a bubble-bursting program 423, learning software 424,and/or the like. These programs are stored in the memory assembly 909 ofFIG. 1A and/or FIG. 1B and are executable by the processing apparatus908 of FIG. 1A and/or FIG. 1B. These programs are described below inmore detail.

FIG. 1B depicts a schematic example of a sensing and display apparatus1300.

In accordance with the example depicted in FIG. 1B, the sensing anddisplay apparatus 1300 includes (and is not limited to)sensory-phenomenon sensors (910, 916) configured to transmit sensorsignals 904 derived from sensory phenomena received from the firstaugmediated-reality space 1000. The sensory-phenomenon sensors (910,916) may also, in one implementation, be configured to transmit sensorsignals 904 derived from sensory phenomena received from the secondaugmediated-reality space 1002.

The sensing and display apparatus 1300 further includessensory-phenomenon effectors (912, 914) configured to transmit effectorsignals 906 associated with sensory phenomena to the firstaugmediated-reality space 1000. The sensory-phenomenon effectors (912,914) may also, in one implementation, be configured to transmit effectorsignals 906 associated with sensory phenomena to the secondaugmediated-reality space 1002. For example, in one implementation, thesensory-phenomenon effectors (912, 914) may be configured to display anyone of a holographic video display, a stereoscopic video display, and/orthe like. The sensory-phenomenon effectors (912, 914) may, in oneexample, include a three-dimensional camera, such as a structured-lightor time-of-flight camera.

In one implementation, the sensing and display apparatus 1300 alsoincludes the interface assembly 902 configured to interface (e.g.,convey, receive) the sensor signals 904 representing sensory phenomenareceived, via the sensory-phenomenon sensors (910, 916), from the firstaugmediated-reality space 1000, and the sensor signals 904 alsorepresenting sensory phenomena received from the secondaugmediated-reality space 1002. The interface assembly 902 may also beconfigured to interface (e.g., convey, transmit) the effector signals906 representing sensory phenomena, via the sensory-phenomenon effectors(912, 914), to the first augmediated-reality space 1000 and to thesecond augmediated-reality space 1002.

In one implementation, the sensing and display apparatus 1300 may alsoinclude the processing apparatus 908 operatively coupled to theinterface assembly 902. In one implementation, the processing apparatus908 is configured to process the sensor signals 904 interfaced with (inassociation with) the interface assembly 902, and to process theeffector signals 906 interfaced with (in association with) the interfaceassembly 902.

By way of example, in various implementations, the sensor signals 904may be derived from one or more of an audio sensory phenomenon, a visualsensory phenomenon, a tactile sensory phenomenon, and/or the like. Invarious implementations, the effector signals 906 may be derived fromone or more of an audio sensory phenomenon, a visual sensory phenomenon,a tactile sensory phenomenon, and/or the like.

In accordance with an option, the sensory-phenomenon sensors (910, 916)and the sensory-phenomenon effectors (912, 914) may further include afirst sensory-phenomenon sensor 910 and a first sensory-phenomenoneffector 912. The first sensory-phenomenon sensor 910 is configured totransmit a sensor signal 904 derived from sensory phenomena from thefirst augmediated-reality space 1000. The first sensory-phenomenoneffector 912 is configured to transmit an effector signal 906 associatedwith sensory phenomena to the first augmediated-reality space 1000.

In accordance with an option, the sensory-phenomenon sensors (910, 916)and the sensory-phenomenon effectors (912, 914) may include a secondsensory-phenomenon effector 914 and a second sensory-phenomenon sensor916. The second sensory-phenomenon effector 914 is configured totransmit an effector signal 906 having sensory phenomena to the secondaugmediated-reality space 1002. The second sensory-phenomenon sensor 916is configured to transmit a sensor signal 904 derived from the sensoryphenomena from the second augmediated-reality space 1002. An example ofthe second sensory-phenomenon sensor 916 includes an eye tracker deviceconfigured to track the pupil of an eye of the user. In oneimplementation, eye tracking is the process of measuring either thepoint of gaze (where the user is looking) or the motion of an eyerelative to the head of the user. An eye tracker is configured tomeasure eye positions and eye movement. Eye movement may be measured ina variety of ways, such as using video images from which the eyeposition is extracted, using search coils, or using an electrooculogram.

For the case where the user wears the sensing and display apparatus1300, the first sensory-phenomenon sensor 910 and/or the firstsensory-phenomenon effector 912 may, in one implementation, face (e.g.,be oriented toward) a direction of a field of view of the user (e.g., inthe first augmediated-reality space 1000). The field of view of the usermay, for example, be a view in an eye forward viewing direction. Forexample, the field of view of the user may include a direction in whichthe user may be able to view the location of the user's fingers,provided the fingers are not positioned out of the user's field of view.For the case where the user is typing on a virtual keyboard, the user'sfingers may be tracked by a first sensory-phenomenon effector 912, evenif the user is looking elsewhere. In one implementation, the secondsensory-phenomenon effector 914 and the second sensory-phenomenon sensor916 face a direction toward (e.g., are oriented toward) the eyes of theuser (the second augmediated-reality space 1002). In someimplementations, one or more of the first sensory-phenomenon sensor 910,the first sensory-phenomenon effector 912, the second sensory-phenomenonsensor 916, and/or the second sensory-phenomenon effector 914 may beoriented in a direction other than facing the field of the view of theuser. For example, sensors and/or effectors associated with auditoryphenomena, and/or sensors for sensing audio and/or inserting audio intoan environment and/or into an ear of one or more users of apparatus1300, may be oriented in a direction of a “field of sound,” e.g., adirection oriented toward one or both ears of a user, away from one orboth ears of the user, omnidirectional, geophonically and/orhydraulophonically coupled to the apparatus 1300 (e.g., touching theskull and/or impedance matched for water when the user is underwater),and/or the like. In another example, sensors and/or effectors may beconfigured to engage peripheral portions of the environment, backportions (e.g., behind the user), blind spots, and/or the like.

In accordance with an example, the interface assembly 902 includes: (A)a first interface module 903 configured to interface with the firstsensory-phenomenon sensor 910; (B) a second interface module 905configured to interface with a first sensory-phenomenon effector 912;(C) a third interface module configured to interface with a secondsensory-phenomenon effector 914; and (D) a fourth interface moduleconfigured to interface with a second sensory-phenomenon sensor 916.Other interface modules for the interface assembly 902 may be added orremoved as may be required and/or desired to accommodate any number ofsensors and/or effectors.

FIG. 1C depicts a schematic example of a sensing and display apparatus1300 in one embodiment.

As depicted in FIG. 1C, the sensing and display apparatus 1300 includes,in one embodiment, a combination of the interface assembly 902 and theprocessing apparatus 908. The first sensory-phenomenon sensor 910, thefirst sensory-phenomenon effector 912, the second sensory-phenomenoneffector 914 and the second sensory-phenomenon sensor 916 are providedas separate items to be integrated with the interface assembly 902,either by the user or by the manufacturer of the sensing and displayapparatus 1300. In accordance with a distributed-control option, theprocessing apparatus 908 is configured to control set up and operationof the first sensory-phenomenon sensor 910, the first sensory-phenomenoneffector 912, the second sensory-phenomenon effector 914 and the secondsensory-phenomenon sensor 916. In accordance with an option, theprocessing apparatus 908 may include instances of a processor unitdedicated for and/or distributed amongst two or more of the firstsensory-phenomenon sensor 910, the first sensory-phenomenon effector912, the second sensory-phenomenon effector 914 and the secondsensory-phenomenon sensor 916. In one implementation, the processingapparatus 908 may be configured to act as a supervising controller whilethe dedicated processors on each of the sensors and on the effectors maybe configured to manage operation of a sensor or an effector (such asdepicted in FIG. 1B).

FIG. 1D depicts a schematic example of a sensing and display apparatus1300.

As depicted in FIG. 1D, the sensing and display apparatus 1300 includes,in one embodiment, a combination of the interface assembly 902, theprocessing apparatus 908, the first sensory-phenomenon sensor 910, thefirst sensory-phenomenon effector 912, the second sensory-phenomenoneffector 914 and the second sensory-phenomenon sensor 916, all combinedand/or integrated as a single unit to be provided to users as such.

In one implementation, expansion slots or room may be provided (and anyequivalent thereof) to accommodate installation of additional sensorsand/or effectors to display apparatus 1300. In one implementation, oneor more sensors and/or effectors may be external to display apparatus1300 and/or interface therewith, such as via one or more wired orwireless communication networks.

With reference to FIG. 1A, FIG. 1B, 1C and FIG. 1D, in accordance withan option, the sensing and display apparatus 1300 (also depicted in FIG.13 and FIG. 14) is configured to scan spatial subject matter and/orthree-dimensional spatial subject matter. By way of suitable sensorassemblies and/or effector assemblies, the sensing and display apparatus1300 is also configured to provide and/or display to a user, an image ofthe subject matter, e.g., as a computer-generated version, acomputer-enhanced version, a computer-mediated version of the subjectmatter, and/or the like (by way of suitable sensor assemblies and/oreffector assemblies) and identified in association with the figures.

In accordance with an option, the sensing and display apparatus 1300 mayinclude a user-wearable interface configured to facilitate a user'swearing of the sensing and display apparatus 1300. For example, in oneimplementation, the sensing and display apparatus 1300 is configured tobe user wearable (such as, worn on the head of the user). It will beappreciated that the other options may be accommodated for the manner inwhich the sensing and display apparatus 1300 is interfaced with theuser. For example, the user may wear a helmet (e.g., hockey, football,and/or the like) to which the sensing and display apparatus 1300 may bemounted. In this way, members of sporting teams may use the sensing anddisplay apparatus 1300, e.g., as a training tool in preparation for asporting game, as an integral part of actual game play, for evaluatingand selecting potential members to be included in a team, and/or thelike.

In accordance with an option, the sensing and display apparatus 1300 isconfigured to be worn in front of the eye of the user. The sensing anddisplay apparatus 1300 may, for example, be mounted to the head of theuser. The sensing and display apparatus 1300 may be configured to record(such as via a camera) a scene available to the eye of the user from thefirst augmediated-reality space 1000 and/or the secondaugmediated-reality space 1002. The sensing and display apparatus 1300may be configured to display and/or superimpose a computer-generatedimage on the original scene available to the eye of the user.

In accordance with an option, the sensing and display apparatus 1300 maybe configured to accommodate one or both eyes of a wearer (e.g., botheyes look through the sensing and display apparatus 1300). In oneimplementation, separate instances of the sensing and display apparatus1300 may be configured to accommodate each eye of the user. The sensingand display apparatus 1300 may be configured to be viewed by one or botheyes of a wearer of the sensing and display apparatus 1300. In oneimplementation, the sensing and display apparatus 1300 may be configuredto provide a two-eye view (e.g., views from two eyes viewing a singlespace, with the same or similar sensors and/or with different and/orcomplementary sensors; views from two eyes, each viewing a differentspace, and/or the like) to a single-eye display within the apparatus1300. Additional views may also be provided to a single-eye display insome implementations.

In accordance with an option, the sensing and display apparatus 1300 isconfigured to provide access through whichever one or both eyes of awearer which may be used to look and see. In one implementation, thesensing and display apparatus 1300 may be referred to as an extramissivespatial imaging digital eye glass, an extramissive spaceglass™ system, aspatial imaging glass, a digital eye glass, a computer eye glass, anEyeTap, a computer-vision system, a seeing aid, a vision aid, and/or thelike. In some instances, the sensing and display apparatus 1300 may bereferred to as a digital eye glass (DEG); it should be understood thatany reference to the digital eye glass does not limit the examples tomerely the digital eye glass, and the examples apply to the sensing anddisplay apparatus 1300. The extramissive spatial imaging digital eyeglass is configured to receive and to transmit light (such as visiblelight) to and from the user or the users of the firstaugmediated-reality space 1000 and the second augmediated-reality space1002.

In accordance with an option, the sensing and display apparatus 1300 isconfigured to: (A) display an image to the user (e.g., to operate as amonitor and/or a display device) in the first augmediated-reality space1000 and/or in the second augmediated-reality space 1002; (B) to intake(e.g., to operate as a camera) an image of the environment in the firstaugmediated-reality space 1000 and/or in the second augmediated-realityspace 1002; and (C) to process (e.g., augment, analyze, modify, and/orthe like) the image displayed to the user in the firstaugmediated-reality space 1000 and/or in the second augmediated-realityspace 1002. To augment the image includes, in one implementation,overlaying computer-generated information (e.g., data and/or images) ontop of the image of the normal world (e.g., the original scene in thefirst augmediated-reality space 1000 of FIG. 1B) the sensing and displayapparatus. The sensing and display apparatus 1300 is configured toaugment and mediate the reality the user perceives in the firstaugmediated-reality space 1000 and/or the second augmediated-realityspace 1002.

In some implementations, user input is by way of self-gesturing (e.g.,by the user) to the sensing and display apparatus 1300. The sensing anddisplay apparatus 1300 may include a computer vision system, such as athree-dimensional computer vision system, which may work with or withouttactile feedback in association with a physical object (e.g., a desktopor similar surface). In one implementation, the sensing and displayapparatus 1300 may also include an auxiliary sensor (such as, athree-dimensional tactile acoustic gesture or a vibration gesture and/oracoustic multimodal gesture input device) in which hitting or strikingor rubbing or touching a surface or object is in view of thethree-dimensional computer vision system.

In one implementation, the program 907 of FIG. 1A may be configured todirect the processing apparatus 908 to execute spatial imagingoperations and/or the sensing and display apparatus 1300 may beconfigured to execute spatial imaging operations by auxiliaryinstructions. The spatial imaging operations may include, in oneimplementation, the holographic display of subject matter, for example,real-time video e real time video chats and/or discussions, such as withthree-dimensional projections of subject matter (e.g., people). As usedherein, the term “spatial imaging” may include, in variousimplementations, any image-scanning device, including, but not limitedto, three-dimensional cameras, three-dimensional sensors, depth cameras,depth sensors, any device configured to use spatial imaging technology,such as: a holographic-imaging device, a three-dimensional imagingdevice, a laser-imaging device, a LiDAR (Light Detection And Ranging)device, a time-of-flight imaging device, a RaDAR (Radio Detection AndRanging) device, a SoNAR (Sound Navigation And Ranging) device, adepth-camera device, a depth-sensor device, a visible light-baseddevice, an infrared-based device, a microwave-based device, asound-based device, and/or the like.

In accordance with an option, the sensing and display apparatus 1300 maybe configured to be miniaturized, and the sensing and display apparatus1300 may be integrated into an existing or new eye glass frame.

In accordance with an option, the sensing and display apparatus 1300 mayinclude other sensors, such as an audio sensor (a microphone and/or anearphone), and/or the like. In various aspects of the sensing anddisplay apparatus 1300, a microphone includes an assembly configured tosense or determine sound pressure, or changes in sound pressure, orflow, or changes in flow, in any medium (solid, liquid, or gas) and anyequivalent thereof.

In one implementation, the sensing and display apparatus 1300 may beused with and/or integrated with a personal computer (such as a desktopcomputer, a laptop computer, a cell phone, a smart phone, a cameraphone, a tablet computer, and/or the like), and may, in oneimplementation, be configured to appear as an eye glass.

FIG. 1E depicts a schematic example of a sensing and display apparatus1300 in one embodiment.

More specifically, FIG. 1E illustrates an example of a sensing anddisplay apparatus 1300 through which one or both eyes of the wearer maysee subject matter; the sensing and display apparatus 1300 may alsoprovide an extramissive vision that may be visible and/or invisible toothers (e.g., other users) in the first augmediated-reality space 1000of FIG. 1A. The first augmediated-reality space 1000 depicted in theexample of FIG. 1E includes a virtual game board of the type used toplay a game of chess. FIG. 1E, FIG. 2A, FIG. 3 and FIG. 4 depictexamples of the sensory-phenomenon sensor (910, 916) and thesensory-phenomenon effector (912, 914) associated with FIG. 1B. It willbe appreciated that FIGS. 1E-1G, FIG. 2A, FIG. 3, FIG. 4 depictsinstances of the effector assembly and the sensor assembly that mayshare the same position on the sensing and display apparatus 1300, andthat this arrangement is for illustrative purposes and for the sake ofclarity.

It will be appreciated that the sensing and display apparatus 1300 maybe simply referred to as a display apparatus or even more simply as anapparatus; generally, the apparatus 1300 may be configured to displayimages (to the user and/or users), may be configured to interface with adisplay assembly, may be configured to sense, and/or may be configuredto interface with a sensing device.

In accordance with an option, the sensing and display apparatus 1300includes, for example, a visible-light receiver 191, which is an exampleof the first sensory-phenomenon sensor 910 (of FIG. 1B) and/or of thesecond sensory-phenomenon sensor 916 (of FIG. 1B), and/or avisible-light transmitter 192, which is an example of the firstsensory-phenomenon effector 912 (of FIG. 1B), or of the secondsensory-phenomenon effector 914 (of FIG. 1B).

In accordance with an option, the sensing and display apparatus 1300includes a head-mountable assembly 183 configured to facilitate securedconnection of the sensing and display apparatus 1300 to the head of theuser. In one implementation, the head-mountable assembly 183 may includea back-of-head band 188 configured to secure the sensing and displayapparatus 1300 to the head of the user, such as may be positioned on theback of the user's head. In one implementation, the back-of-head band188 may include a sensor configured to sense the zone behind the user'shead. The sensor located on the back-of-head band 188 may, for example,be shared with other users associated with the first augmediated-realityspace 1000 of FIG. 1A.

In one implementation, the sensing and display apparatus 1300 includesthe digital eye glass 180 including various components such as anorientation and inertial measurement unit 184, a LiDAR unit 187, avision system 193, an optical sensor 100, a display unit 181, aremovable shade 110, and/or the like. In one implementation, theremovable shade 110 may be a two inches by 4.25 inches (approximately 51millimeters by 108 millimeters) standard size welding shade. Theremovable shade may allow the display unit 181 to be visible in brightsunlight (e.g., using ANSI Shade numbers 5 to 7) or on a bright butcloudy day (e.g., using ANSI Shade numbers 2 to 4), or for using thesensing and display apparatus 1300 for welding (e.g., using a darkershade), while seeing the electric arc using HDR (High Dynamic Range)imaging.

In accordance with an option, the digital eye glass 180 is configured toinclude a singular glass through which both eyes see, such as in theconfiguration of a welder's helmet. In accordance with a variation, aseparate display or mediation zone may be provided for each eye, withinthe one glass, or a separate glass for each eye, or a monocular glassfor one eye.

In one implementation, the digital eye glass 180 may be configured tofunction as a seeing aid, a vision aid, and/or the like, and may beconfigured to provide high dynamic range (HDR) vision, e.g., in whichthe wearer may see in complete darkness while also looking into anelectric arc, or looking into bright car headlights in a dark alley andstill being able to clearly see the car's license number and thedriver's face.

In accordance with an option, the removable shade 110 is electrochromic,and is configured to be controllable by the optical sensor 100 and/orthe LiDAR unit 187 and/or the vision system 193 and/or by anycombination and permutation thereof. For this case, the digital eyeglass 180 is configured to adapt to a wide range of vision conditions,such as change from indoors to outdoors, sun, cloud, electric arcwelding, bright lights, etc., as might occur, or provide protection fromdeliberate or lighting attacks such as from laser pointers or othersources of lights, to which the digital eye glass 180 may afford eyeprotection for the user.

In accordance with an option, the optical sensor 100 is located aroundthe periphery of the removable shade 110 of the digital eye glass 180 orincorporated directly therein. The optical sensor 100 may also, oradditionally, be located above the removable shade 110. When theremovable shade 110 is removed, this may be considered as Shade 0 (e.g.,the setting of the removable shade 110 becomes Shade 0).

In accordance with an option, the sensors are positioned above theremovable shade 110. In accordance with an option, the sensors includean infrared transmitter 186 and/or an infrared receiver 185. In oneimplementation, an infrared receiver 195 is installed to the digital eyeglass 190. The infrared receiver 185 may, in some implementations, bereferred to as an infrared detector. The infrared transmitter 186 andthe infrared receiver 185 may, in one implementation, be configured tocooperate as a range-sensing vision system, such as an infrared LiDAR orother three-dimensional infrared camera system, and/or the like.

In accordance with an option, a visible-light receiver 191 is providedand may, in one implementation, be configured as a visible-light camera(e.g., for providing photographic images having color), and may, in oneimplementation, be assembled together with a range map or range imageformed by the infrared transmitter 186 and the infrared receiver 185.

In accordance with an option, a visible-light transmitter 192 isprovided, and is configured to operate in the visible-light spectrum toilluminate subject matter, such as the surface 130 (e.g., a tabletop)with visible content such as an advertisement, a sponsor notice, acounter, a time/date indicator, or other indicia such as a projection170, which is visible to wearers of the digital eye glass 180. In oneimplementation, such projection may also be visible as well as to thenaked eyes 198 of any users (persons) not equipped with their instanceof the digital eye glass 180.

In one implementation, the sensors and the effectors used in the sensingand display apparatus 1300 may be fixedly attached and/or mounted to aperipheral frame (which is an example of a frame assembly) positioned(e.g., on the head of the user) around the eyes of the user. The digitaleye glass 180 affords (for two eyes) a spatial image that may, in someimplementations, have a horizontal-parallax-only image and/or astereoscopic image. This may be done, for example, using two separateinstances of the display unit 181 or a stereo instance of the displayunit 181 in a single glass through which both eyes of the user may see,or in separate glasses. The interpupillary distance (IPD) values fromthe 1988 Army Survey indicate an average of 64.7 millimeters (mm) formen, and an average of 62.3 millimeters for women. If the average iscomputed for these two (e.g., for both genders), the result is a totalof 64.7+62.3=127.0 millimeters and dividing by two gives a result of127/2=63.5 millimeters and is equal to is 2.5 inches. Thus, theinstances of the display unit 181 may be positioned, for example, about2.5 inches apart at their eye points. In some implementation, instancesof display unit 181 may be positioned adjustably and/or dynamically,such as based on a prior, concurrent and/or real-time measurement of theIPD of a given user. For example, in one implementation, a sensordirected toward the user's face may detect and/or measure an IPD for theuser, and the distance between display units 181 for each eye may beadjusted on that basis. In another implementation, an active displayarea of each display unit 181 may be adjusted based on a measured userIPD, e.g., so as to present the displayed content in appropriate regionsof the display area based on particular user IPD.

In accordance with an option, the sensing and display apparatus 1300 mayinclude the digital eye glass 180 configured to provide a two-eyeddisplay, a monocular display, and/or the like. In the case of a two-eyedview, there is provided an oculus dexter point-of-eye 199A, and anoculus sinister point-of-eye 199B (that is, the right EyeTap point andthe left EyeTap point).

In one implementation, the capacity to capture true (or near true)three-dimensional images may be used to create a lightspacecollinearizer configured to create a synthetic effect for the sensingand display apparatus 1300. For example, the rays of eye-ward boundlight may be diverted through an imaging system, processed, andre-rendered to be substantially collinear with the original rays. Thelightspace collinearizer (which may, in some implementations, bereferred to as a synthetic collinearizer) may be configured to capture atrue or near true three-dimensional model, and to calculate the imagethat would have arisen by a camera as if the three-dimensional modelwere actually to have been placed inside the wearer's view with thecenter of the lens (iris, nodal point, optical center, or the like) ofthe camera at the center of the lens of the eye of the user.

In one implementation, a three-dimensional camera may be used and/orconfigured to synthesize the effect of a spatial imaging device, and mayinclude a lightspace-analysis glass, passing over the wearer's eyes; forexample, the entire visor of the sensing and display apparatus 1300 maybe configured to act as a lightspace-analysis glass. In oneimplementation, the three-dimensional camera may be configured togenerate the lightspace, lightfield, plenoptic, holographic, and/or thelike image of all rays of light passing through the sensing and displayapparatus 1300. The processing apparatus 908 of FIG. 1A may beconfigured to be responsive to an output of the (e.g., virtual)lightspace-analysis glass, and the processing apparatus 908 may beconfigured to compute rays for a lightspace synthesis glass. In such animplementation, the combination of a three-dimensional camera, aprocessor, and a display may embody a lightspace collinearizer.

Because of the true three-dimensional nature in implementations of thevision system, the digital eye glass 180 and the digital eye glass 190may each be configured to capture a true three-dimensional image of thepublic subject matter 140, such as a shared gaming table (e.g., a realor imagined or virtual chessboard, or the table surface 130), and renderthis real view, or a computer-modified view of the table 130, or anycombination of these, as if the image were captured by a camera wherethe camera itself, a lens of the camera, a film of the camera, a CCD ofthe camera, and/or the is located exactly at and/or substantially nearthe center of the lens of the oculus dexter (right eye), as well asanother view as if the image were captured by a camera located exactlyat and/or substantially near the center of the lens of the oculussinister (left eye) of the wearer of the sensing and display apparatus1300. CCD stands for charge-coupled device.

In one implementation, each of the digital eye glass 180 (for the firstuser) and the digital eye glass 190 (for the second user) may use itsown vision system plus information received from the digital eye glass190 to construct an even more detailed, true and/or accuratethree-dimensional model of reality than its own vision system alone. Inthis sense, in one implementation, a number of different participantsmay share a computer-mediated reality in which the real (physical) world(the first augmediated-reality space 1000) is captured with relativelyhigher detail, precision, and/or accuracy, and/or including dataassociated with additional and/or supplementary perspectives,directions, views, and/or the like.

In accordance with an option, the sensing and display apparatus 1300 maybe configured to operate in a multiplexed manner, and/or may beconfigured to execute multiplexing. For example, the digital eye glass180 and the digital eye glass 190 may be configured to cooperate, suchas by time-division multiplexing, and thereby alternately illuminatingthe scene to sense and/or understand the objects located in the firstaugmediated-reality space 1000. The multiplexing may be effectuated, forexample, by using code-division multiplexing (e.g., using differentspreading sequences or spread spectrum or spread spatializationpatterns), and/or by collaborative sensing. In the latter case, thedigital eye glass 180 and the digital eye glass 190 are configured towork together to illuminate and sense the public subject matter 140 andthe surface 130. For example, while the infrared transmitter 186illuminates the scene (the first augmediated-reality space 1000) withinfrared light, the infrared receiver 185 and the infrared receiver 195both sense the reflected infrared light reflected from physical objectsin the scene (the first augmediated-reality space 1000). Thecross-sensing between the digital eye glass 180 and the digital eyeglass 190 provides additional scene information through the extremeparallax that exists owing to the longer baseline between the digitaleye glass 180 and the digital eye glass 190 in the firstaugmediated-reality space 1000.

In one implementation, the program 907 of FIG. 1A (used in an instanceof the sensing and display apparatus 1300) is configured to execute themultitasking operations with other instances of the sensing and displayapparatus 1300.

The result, in one implementation, is a Synthetic Aperture LiDAR, or thelike, having a resolving power that may be mathematically broken downinto a separate small-parallax inter-glass baseline, and alarge-parallax intra-glass baseline.

In accordance with an option, the digital eye glass 180 may include acomputer 160 (which is an example of the processing apparatus 908 ofFIG. 1A). The computer 160 may be either built into (integrated with)the digital eye glass 180 or may be separate from the digital eye glass180, and/or may be configured, for example, to fit in a shirt pocket ofthe user. In accordance with an option, the computer 160 includes anetwork connection 150, such as a wireless connection, WLAN (WirelessLocal Area Network), a WiFi™ network, PSA (Personal Area Network), aBluetooth™ network, a cellular connection, CDMA (Code Division MultipleAccess), HSPA (High Speed Packet Access), HSDPA (High-Speed DownlinkPacket Access), GSM (Global System for Mobile Communications), and/orthe like. For some implementations where the computer 160 is used inconnection with the digital eye glass 180, the computer 160 may beintegrated with the geophone 182.

In accordance with an option, the digital eye glass 180 isself-contained. In accordance with an option, the digital eye glass 180is tethered to the computer 160, and the digital eye glass 180 may beplaced in a pocket of a shirt of a wearer, such as for storage. For someimplementations where the digital eye glass 180 is tethered, the wiringmay be concealed inside cloth pipes, cloth tubing, and/or the like,having the appearance of ordinary eyeglass safety straps. For example,the eyeglass safety straps may include a clip 189 configured to securelyclip onto an article of clothing of the user to provide strain relief ofthe eyeglass tether. By way of example, the clip 189 may be a “crocodileclip”, an “alligator clip”, a spring clip, a spring-loaded clamp of thekind that are used for headsets, lavalier microphones, and the like,e.g., to grip the wearer's clothing, thus providing, for example, a formof strain relief.

In some situations, the computer 160 (which is an example of theprocessing apparatus 908 of FIG. 1A) may be placed on the floor or tablein an office or home space when the user is positioned in a seatedposition. For some such implementations, the computer 160 may be housedinside a sensor-interface unit 120 (e.g., a tabletop unit), whichincludes the geophone 182 and in this way the sensor-interface unit 120is geophonic enabled. The sensor-interface unit 120 includes a sensorinterface assembly configured to interface the sensory inputs to beprovided to the computer 160. In one implementation, thesensor-interface unit 120 is configured to listen geophonically to soundvibrations in the table surface, such as when chess pieces are slidacross the table (or touch the table), when someone taps their finger onthe table, and/or the like.

The term geophone may, in various implementations, refer to any of avariety of pressure transducers, pressure sensors, velocity sensors,flow sensors, and/or the like that convert changes in pressure,velocity, movement, compression, rarefaction, and/or the like, such asin solid matter, to electrical signals. Geophones may includedifferential pressure sensors, absolute pressure sensors, strain gauges,flex sensors on solid surfaces like tabletops, and the like. Thus, ageophone may have a single listening port or dual ports, one on eachside of a glass or ceramic plate, stainless steel diaphragm, or thelike, or may also include pressure sensors that respond only to discretechanges in pressure, such as a pressure switch which may be regarded asa 1-bit geophone. Moreover, the term geophone may also be used inreference to devices that only respond to changes in pressure orpressure difference, i.e. to devices that cannot convey a staticpressure. In one implementation, the term geophone is used to describepressure sensors that sense pressure or pressure changes in anyfrequency range whether or not the frequency range is within the rangeof human hearing, or subsonic (including, in one implementation, all theway down to zero cycles per second) or ultrasonic. Moreover, the termgeophone may be used, in some implementations, to describe any kind ofcontact microphone or similar transducer configured to sense vibrationsor pressure or pressure changes, such as in solid matter. Thus, the termgeophone may be used in reference to contact microphones that work inaudible frequency ranges as well as other pressure sensors that work inany frequency range, not just audible frequencies. A geophone may beconfigured, for example, to sense sound vibrations in a tabletop,scratching, pressing downward pressure, weight on the table (e.g., viaDC or Direct Current offset), as well as small-signal vibrations (e.g.,via AC or Alternating Current signals). In some implementations, theterm “Natural User Interface” may be used in reference to this and othersimilar forms of interaction using physical media such as real-worldobjects and eliminating the indirection of metaphors. In accordance withan option, the digital eye glass 180 embodies a Natural User Interface,particularly with the assistance of the sensor-interface unit 120.

In addition, in one implementation, the LiDAR unit 187 and the LiDARunit 197, which are configured to sense the three-dimensional objectpositions, and the like, provide data to the computer 160 along with thesensor-interface unit 120, to provide multimodal sensory tactilefeedback by way of the geophone 182 and other sensors.

In accordance with an option, the geophone 182 in the sensor-interfaceunit 120 is configured to transmit and receive and may be fixedlyattached to a tabletop to provide transmission of vibrotactile energy tothe tabletop that may be felt by the user, such as a click or touch ortaction by a user's fingers. Alternatively, in one implementation, thisforce and/or vibrotactile energy may be used semi-destructively, such asfor dramatic effect.

For example (as depicted in FIG. 1E), in a game of “augmediated chess”,the geophone 182 may be configured to shake the table and knock overreal chess pieces when one player wins the game, thus creating adramatic visual effect even visible to the naked eye 198 of anon-glass-wearing participant (e.g., a user who does not use the sensingand display apparatus 1300).

Moreover, if multiple participants set their respective instances of thesensor-interface unit 120 and the computer 160 on a surface 130, such asa tabletop, there may be multiple instances of the geophone 182 and thegeophone 194 in a respective instance of the sensor-interface unit 120and the computer 160, such as to form an array of geophones. An inertialmeasurement unit 184, accelerometer, and/or the like in each instance ofthe sensor-interface unit 120 and the computer 160, and/or otheradditional sensors may be used in one implementation to compute therelative position of the sensor-interface unit 120 and the computer 160,such as to form a phased array of sensors that may localize acousticand/or other physical disturbances in the surface 130.

The digital eye glass 180 and the digital eye glass 190 are, in someimplementations, each configured to sense and/or emit light, such asvisible light. In implementations of the fully bidirectional instances,the digital eye glass 180 DEG may be configured to sense and emit lightin both directions, affording the following: extramissive active visionforward-looking, and extramissive active vision inward-looking, such asto the eye itself. This may include eye-tracking (e.g., as sensing) anddisplay (effecting), as well as spatial imaging of the environment(e.g., as sensing) of the first augmediated-reality space 1000 of FIG.1A, and projection to the environment (e.g., as effecting). In thissense, the digital eye glass 180 sends and receives light to and fromthe eye, and the digital eye glass 180 also sends and receives light toand from subject matter in view (e.g., of the eye of the user).

In accordance with an option, the digital eye glass 180 is configured tosense the environment around the user and/or to sense the user. Thespatial-imaging sensor assembly, such as the LiDAR unit 187, may, in oneimplementation, be configured to provide a depth map 1308 (depicted inone example in FIG. 13), such as to determine the spatial coordinates,positions, orientations, and/or the like of the user's arms, hands,fingers, and/or the like. The spatial-imaging sensor assembly may alsobe configured to provide a depth map to determine the spatialcoordinates, positions, orientations, and/or the like of the user'slegs, feet, and/or the like, such as for the case where the user looksdown at the ground. In one implementation, the two kinds of scannablesubject matter may be separated out, for example, as non-self subjectmatter (e.g., other objects in the room), and self. When the two meet,the meeting may be detected and/or tracked, and the situation may bereferred to in some implementations as “physical contact” or “taction”.

In accordance with an option, the processing apparatus 908 of FIG. 1Amay, in one implementation, include a taction detector. The tactiondetector is configured to detect taction (e.g., physical contact betweenthe human body and another object, between two objects, and/or the like)and responds accordingly. The taction detector may, for example, includeprogram instructions configured to detect the occurrence of taction. Inaccordance with an option, the taction detector includes the geophone182 configured to pick up and/or otherwise detect the sound of taction,and the vision system (the sensing and display apparatus 1300) isconfigured to observe the occurrence of taction by three-dimensionalscanning of a surface (such as a tabletop), and determining when and ifthe hands, fingers, and/or the like touched the surface. In oneimplementation, this may be accomplished via a point cloud, and when thepoint cloud of the hands or the fingers intrude upon the homography ofthe plane, such as manifested by the group action, (1). Formula {1} maybe used by the processing apparatus 908 of FIG. 1A, in oneimplementation, to detect the occurrence of taction by a tactiondetector 400, such as depicted in one example in FIG. 1AA.

$\begin{matrix}{{f(x)} = \frac{{Ax} + b}{{c\; \dagger \; x} + d}} & {{FORMULA}\mspace{14mu} \{ 1 \}}\end{matrix}$

Any two images of the same planar surface in space are related by ahomography (assuming a pinhole camera model). This allows many practicalapplications, such as image rectification, image registration,computation of camera motion (rotation and translation) between twoimages, and/or the like. Once camera rotation and translation have beenextracted from an estimated homography matrix, this information may beused, such as for navigation, to insert models of three-dimensionalobjects into an image or video, and/or the like, so that they arerendered with the correct perspective and appear to have been part ofthe original scene.

In one implementation, [A] is a 2 by 2 (2×2) linear operator, [b] is a 2by 1 (2×1) translation, [c] is a 2 by 1 (2×1) chirp (projective), [d] isa scalar constant, and [x] is a 2 by 1 (2×1) spatial coordinate, and[x]=[x1,x2]^(T), with T indicating transposition.

In one implementation, a statistical significance test may be performedupon the degree of homographic intrusion detected by the tactiondetector. An apparatus configured to perform a measure of the degree ofhomographic intrusion may, in some implementations, be referred to as ahomography intrusion estimator or a homographic-intrusion estimator. Inan implementation where such an estimator is thresholded, used as atrigger for another action, and/or the like, the apparatus may bereferred to as a homography intrusion detector. In one implementation,detection and estimation theory, such as Neyman-Pearson theory, or thelike, may be applied to a homography intrusion detector 401 depicted inFIG. 1AA.

For the case where a combination of vision, audition (e.g., through thegeophone 182), and/or other sensory input is employed, a confluencesensor 402 (depicted in FIG. 1AA) may be used to fuse these two modes ofsensing.

A confluence sensor may, in one implementation, may be built from acombination of a visual trusion sensor and/or an acoustic trusionsensor, such as shown in the example of FIG. 1EE. In one implementation,such sensors may use a neural network configured to estimate trusiveness(intrusiveness, extrusiveness, or both), taction, and/or the like.

In one implementation, the digital eye glass 180 may be supported on thehead (of the user) by a frame, such as with the head-mountable assembly183 (referred to in some implementations as a head strap). In oneimplementation, the head-mountable assembly 183 may be split so that aportion of (e.g., half of) the head-mountable assembly 183 is aligned,positioned, and/or the like below the occipital lobe, another portion(e.g., the other half) is aligned, positioned, and/or the like above theoccipital lobe, thus making the digital eye glass 180 stay on the headof the user during intense physical activity such as gymnastics, yoga,and/or swimming where water currents might otherwise brush the digitaleye glass 180 away from the face of the user. For instance, a frontplanar surface of a sealed (water tight instance) of the digital eyeglass 180 may be configured to operate above and/or below water.

Aspects of the disclosed apparatuses, methods and systems may include avisual intrusion detector, which detects intrusion into a space(plane/homography, spherical object, or the like), or more generally, atrusion estimator, which can estimate the degree of trusion (eitherintrusion or extrusion, or both simultaneously), and, subsequently (ordirectly) detect trusion by way of some kind of threshold, or the like(e.g., gesture sensing, estimation, detection, and gestureclassification).

In some implementations, trusion sensors can also be confluence-based,e.g., a result of a confluence of multimodal trusion sensing. Examplesinclude a trusion sensor that senses by way of electromagnetic radiation(a first mode) and acoustic radiation (a second mode).

FIG. 1EE depicts a signal-processing system in one embodiment thataccepts input from a light sensing or light sensing and effectory(luminary) apparatus, as well as from an acoustical-sensing apparatus,or an acoustical sensing and effectory (e.g., sonar) apparatus.

FIG. 1EE illustrates a confluence sensor, including a visual trusioninput 1EE20 and an acoustic trusion input 1EE10. In one implementation,there are two kinds of trusion: intrusion (as we observe when, forexample, a burglar or “intruder” enters into a premises) and extrusion(as we observe, for example, when an aluminum “I” beam is made by“extruding” metal in the shape of the letter “I”).

Intrusion means, in one implementation, to “thrust” or “push” into oronto or against something, or to enter into the vicinity of something.Intrusion may be binary on/off, e.g., “triggers” when an intruder entersa premises, and/or can occur by degrees. For example, one can intrudethe personal space of another by getting close to a person's “personalbubble”, e.g., without even touching the person.

We can intrude onto the surface of a table, e.g., to draw virtualobjects on the surface of the table, by, for example, coming close tothe table, hovering over it, touching It, and/or the like. We can pushagainst a table lightly. We can also push harder against the table.Thus, in one implementation, the degree of intrusion can vary fromcoming near the tabletop, to touching the table top very lightly, topressing lightly, to pushing, to pushing very hard, emphaticallypounding the table with one's fist, and/or the like. Thus, the systemcan sense trusion and taction (e.g., tactile elements like touch,tapping, rubbing, hitting, and the like).

We can also pull away from the table (extrusion). A wearable visionsystem such as a visual trusion sensor 1EE30 can sense touch with thetable, the degree of intrusion or extrusion, and/or the like. In oneimplementation, a touch sensor may be configured to engage multimodalsensing by a plurality of modes, such as, for example, also sensingusing an acoustic trusion sensor 1EE25.

As used herein, an acoustic signal may include a sound level all the waydown to DC (zero cycles per second). In one implementation, the acoustictrusion sensor 1EE25 may be a strain gauge in or on the table, whichsenses flexion of the table, and/or a “bender” or “bend sensor” in thetable, and/or sensors on the legs of the table that sense weight orforce upon the table.

In one implementation, the acoustic trusion sensor 1EE25 and the visualtrusion sensor 1EE30 are configured to supply trusion metrics and/or amultidimensional trusion signal 1EE35 that may be considered a featurevector of a neural network formed by the node 1EE40, the node 1EE50, thenode 1EE70, or the like, together with connection by “neurons” orweights or elements. The connection 1EE45 and the connection 1EE60, orthe like, together with the nodes, form a neural network that is trainedto recognize intrusion, touch, and extrusion.

In one implementation, the taction signal 1EE80 detects and/or providesa degree of intrusion, which may be configured as a singlefloating-point number, a feature vector richly capturing the essence ofthe intrusion, and/or the like.

In one implementation, the taction signal 1EE85 captures a degree oftouch, and conveys information about touch, such as how much touch thereis. In one implementation, it conveys information about how many fingersare touching, which fingers are touching, and/or the like in relation toa real physical object such as a tabletop or the like, or a virtualobject such as a shared virtual object that one or more people canexperience together.

In one configuration, the taction signal 1EE80, the taction signal1EE85, and the taction signal 1EE90, respectively, passes into theintrusion detector 1EE91, the touch detector 1EE92, and the extrusiondetector 1EE93. Detected signals, respectively, the signal 1EE94, thesignal 1EE95, and the signal 1EE96, are supplied to a gesture sensor1EE97 which senses the gesture being performed in association with areal or virtual object upon which taction and/or trusion are to besensed.

The gesture sensor 1EE97 may provide various output signals 1EE98, suchas where each is indicative of a particular gesture.

Intrusion sensing may be performed against various objects, real and/orvirtual. For example, intrusion may be sensed with respect to planarand/or approximately planar surfaces like walls, tables, floors,building faces, and the like. Thus, a homography (algebraic projectivegeometry on a planar surface, under for example, the orbit of theprojective group of coordinate transformations) can be defined on aplane, and an intrusion detection performed thereupon, as shall beillustrated, by way of example, in the context of FIG. 2B. Intrusiondetection thereupon provides information about gestures that operateupon this planar surface.

In one implementation, any manifold may be configured as a surface uponwhich users can interact, as a “string” in space upon which users caninteract, and/or the like.

Accordingly, the trusion and taction sensor of FIG. 1EE may beconfigured as a manifoldizer-based interactor (the “Interactor”) forwearable interaction and interaction design through the toposculpting™system (e.g., topological sculpting).

Apparatuses, methods and systems for interaction, or interaction designcomprise, in some embodiments, the creation or editing of a virtualtopological space, topological sculpting or the toposculpting™ system,that is locally homeomorphic to Euclidean space, or approximately so,thus going beyond homographies of rigid planar patches to also includeinteraction with other objects like spheres, virtual bubbles, polygons,pipes, curved pipes, lights, points, icons, letters, irregular shapes,curved lines and surfaces, blobs, and the like.

(An n-dimensional topological manifold is a second countable Hausdorffspace that is locally homeomorphic to n-dimensional Euclidean space.)

For example, a rope or wire or cable held or visualized in 3-dimensional(3D) space may be represented as a second countable Hausdorff space thatis locally homeomorphic to a 1-dimensional Euclidean space.(Colloquially, it “behaves like a line in a small neighborhood of apoint along curve” defined therein.) In one embodiment, this forms abasis for spatial interaction in an AR environment. AR stands forAugmediated Reality.

The term “manifold” may be used herein in a wide and non-limiting sense,e.g., to include partial manifolds, such as a “figure-8” shape which isnot a manifold but includes pieces of manifolds “stuck together” beyondthe point of intersection (i.e. the point of “non-manifoldness”).

Likewise, interaction with a folded subway map falls within the scope ofimplementations of the disclosed apparatuses, methods and systems,whether or not it contains portions (e.g., sharp folds) that aremathematically less well-behaved. Interaction with a computer screen(real or virtual) is also included in some implementations, regardlessof overall topological or non-topological structure(s).

As a second example of a manifold interaction, Alice, a woman withautism, enters a subway station and picks up a map. She is confused byall the lines on the map, but her Spaceglass (spatial imaging glass)assists her by, in one implementation, using its “holographic vision”that can “see” and “understand” the map and help her navigate it. TheSpaceglass recognizes that it is looking at a surface (e.g., a secondcountable Hausdorff space that is locally homeomorphic to a2-dimensional Euclidean space, i.e. the surface behaves like a plane ina small neighborhood of a point on the paper surface).

In one embodiment, the Spaceglass™ system may achieve this by way of anextramissive spatial imaging system that emits light toward objects inthe environment around it, and then senses the returned light, as wellas ambient or natural light. It can include a lock-in camera thatdistinguishes (using Lightspace) spatializations that are due to lightemitted from it, from light that returns from elsewhere (e.g., ambientlight).

The Spaceglass may thus be configured to distinguish auto light sourcesfrom allolight sources (Greek prefixes “auto” meaning “self”, i.e. lightdue to illumination from the Spaceglass itself, and “allo”, meaning theopposite of self, i.e. light due to sources other than the Spaceglass).

In one implementation, the device or mechanism that allows thisdistinction is called a “Lightspacer”. A lock-in camera is oneembodiment of the lightspacer, but it might also include, in someimplementations, a timing circuit that flashes the light brightly onevery second frame of video and compares autolumination images withallolumination images, as distinct lightvectors, or the like, using thetheory and praxis of lightspace.

Alice's Spaceglass begins to recognize some of the subway routes shownon her map. Although the map is folded, and even the parts that areseemingly “flat” are somewhat curved, due to sagging of the map, herSpaceglass can sense that the map defines a locally 2-dimensionalsubspace of the 3-dimensional space it is in. Moreover, the subwayroutes themselves are manifolds: one-dimensional manifolds within thetwo-dimensional surface of the map: itself a manifold in thethree-dimensional space around it.

As she touches the map, her Spaceglass recognizes the touching gesture,according to the taction/trusion sensor of FIG. 1EE. A contact-sensor ortouch sensor in the Spaceglass, or its processor, senses touch andcontact with the map's surface, generating a taction signal, as well ascontact with the one-dimensional manifold of a subway route, alsogenerating another taction signal. The taction signal 1EE80, the tactionsignal 1EE85, and the taction signal 1EE90 work together, i.e. therebeing two instances of the taction/trusion sensor of FIG. lEE, oneresponsible for sensing taction with the paper of the map, and the otherresponsible for sensing taction/trusion with the subway route shownthereupon. The first sensor senses taction/trusion (“tusion”) with thetwo-dimensional manifold of the paper, whereas the second senses tusionwith the one dimensional manifold of the subway map drawn on that paper.

The touch-sensor may, in various implementations, include one or bothlevels of touch sensing: first when the paper is touched, and secondwhen the path, or curve, or other manifold on the paper is touched. Ineach case, a response may be generated in the form of an augmediateddisplay, or display media, rendered as if emanating from a manifold suchas the paper surface or subway route thereupon.

In addition to reading the map in an augmediated sense, she can alsopropose a design for a new subway by manipulating the design and movingstops around along the one-dimensional manifolds as if they were beadson a wire. As she touches a stop, it may be configured to light up withinformation about the stop rendered as if upon the page of the map. Thenwith a particular gesture, such as a “pinching” gesture (e.g., a sensingof when her thumb and index finger touch, after approaching a subwaystop or other map point from opposite directions), the selected objector point-of-interest may be highlighted in the virtual space and can beslid back and forth along the submanifold of the subway route. This may,in one implementation, be called a “planifoldizer”, e.g., a manifoldplanning system that allows a manifold to be “edited” or “sculpted”(“toposculpted”) in real time, as a form of interaction design.

In one embodiment, the planifoldizer may combine tactile feedback of areal physical object, like a piece of paper, with an editable virtualmanifold.

In one implementation, the planifoldizer may be configured with adimensional hierarchy. Sliding the thumb and forefinger along the sheetof paper is recognized as a gesture that can select a point in adifferent way than doing so in 3D space off the page. Thus, thedimensional hierarchy recognizes gestures that are confined to the 2Dmanifold as different than those in 3D (three dimensional) space not on2D (two dimensional) manifold. Moreover, another class of gestures isthose confined to the one dimensional (1D) manifold, e.g., running thethumb and forefinger along the subway route, to approach a particularsubway stop from either side, the thumb on one side and the forefingeron the other side, and then bring the thumb and forefinger together toselect a particular stop. This gesture, for example, invokes amove-along-the-route operation, whereas doing the gesture in the 2Dpaper plane but off the route invokes a move-the-route operation (tomove the whole route and not just the stop).

In some implementations, the planifoldizer can be used for routeplanning on any map or even on a blank sheet of paper which paper isused as a user-interface to a planifoldizer to give an otherwise “airy”user-interface the power of tactile feedback. Such a paper sheet, inthis context, may be called a “planifold” (interactive manifold forplanning a route, for example).

In one embodiment, a manifoldizer synthesizes an interactive inputdevice in which the user is invited to generate a manifold in virtualspace. Examples include a gesture-controlled string generator, and agesture-controlled surface generator.

In some embodiments, through gestures, the user creates a manifold thatfollows approximately the time-integral of the user's gesture along atangent-line of the manifold to thereby define it. For example, avirtual string, rope, or wire, is generated by moving a finger throughspace, while the manifoldizer synthesizes a time-integral of themovement. The result is akin to a long-exposure photograph as if a lightbulb were moved through it (accumulative lightvectoring). This form oftoposculpting may be referred to as the abakosculpting™ system.“Abakosculpting” is from the Hebrew form “abaq”, meaning “dust”—the sameroot word from which the word “abacus” is derived. In some sense,abakosculpting is like working an abacus, i.e. moving points-of-interestalong virtual ropes, strings, wires, or the like, as if they were beadson an abacus. The abakography™ system can also be referred to as theDusting™ system, and it can be said that we “Dust” an object when we useit as a guide to construct an abakograph of the object, or of a rouletteinvolving the object, or otherwise involve the object in the making ofan abakograph. One method of dusting is to sprinkle or throwretroreflective powder in the air, and blow it around, while a cameraand projector sharing a common optical axis illuminate this dust. In oneembodiment of “Dusting”, a long exposure video summation, the “dust”traces out streaks of light that may be regarded, metaphorically, as thewires on an abacus. In another embodiment, some of the powder isdeposited on a user's hand so that hand gestures generate abakographs.In another embodiment the powder is not required because the computervision algorithms are sufficiently advanced as to be able to simulatethe process using computer vision alone.

In another example, a two-dimensional manifold is generated inthree-dimensional virtual or augmediated space, as a tangent surface toa user's hand or other body part.

Such devices may be referred to as “input manifoldizers”.

In one embodiment, a manifold is displayed in virtual or augmediatedspace, or by way of an augmediator, apparatus, or the like, and the usercan interact with the pre-displayed manifold.

Such devices may be referred to as “output manifoldizers” or“manifoldized displays”.

In one embodiment, an input manifoldizer may be used to generate anoutput “manifoldizer”.

Examples include the generation of a 1-dimensional manifold display bymoving the finger through space, resulting in something having thegeneral appearance of “light rope” or a string of LEDs (Light EmittingDiodes), that grows longer as the finger moves more through space. Theresulting “light rope”-like manifoldizer is a one-dimensional manifoldsynthesized in three-dimensional space, which is interactional in thesense that it behaves as would a touch-rope.

In one implementation, objects may be synthesized along the 1D manifoldas if beads on a string, for example. These objects correspond, forexample, to subway stops along a subway route. In this sense, we callthese objects “spats”, as per the British unit of solid angle (1 spat isequal to 4 pi steradians of solid angle). A spat can be selected orcreated or deleted along the path of the 1D manifold. Spats may also, inone implementation, correspond to real-world panoramas when desired.Clicking a spat along the virtual string may, for example, bring up apanoramic vista of a particular subway stop. A subsystem of thedisclosed apparatuses, methods and systems, which we call theinteractor, senses gestures and synthesizes responses due to thosesensed gestures. It can distinguish between gestures acted along the 1Dmanifold, as compared with gestures acted in the plane of the page butnot along the 1D manifold, as compared with gestures acted outside theplane. Thus, grasping an object from within the page, for example, givesa different result than grasping it from off the page (i.e. approachinga spat from off-page, such that the initial point-of-contact with thepage is the spat itself, versus touching the page elsewhere and thensliding to the spat).

In various embodiments, other gestures may be used and alsodisambiguated. For example, in one implementation, touching the page onthe 1D manifold but away from the spat will select the manifold, andthen allow a constrained-slide along that manifold. Touching the pageelsewhere allows a slide of the manifold itself, or a slide to themanifold—a distinct gesture from the first one.

In the next dimension, the input/output manifoldizer behaves, in oneimplementation, as would a touch-screen, being therefore atwo-dimensional manifold (locally Euclidean 2-dimensional space)synthesized in three-dimensional space.

The disclosed apparatuses, methods and systems may also be configuredfor higher dimensions, e.g., a hypercube display in a higher dimensionalembedding space synthesized as a manifoldizer-based interactor.

In some embodiments, the manifoldizer may be configured for visualinteraction design (e.g., virtual screen interaction, or sculpting3-dimensional objects), but the disclosed apparatuses, methods andsystems are not limited to visual interaction design. For example,embodiments may be configured for manipulation of other media such assound files (e.g., as one-dimensional manifoldized waveforms) or othermedia such as symplectic and metaplectic manifolds for interaction withphase spaces of Hamiltonian and classical Lagrangian mechanics. Examplesinclude manipulation of metaplectomorphisms of the Time-Frequency plane(e.g., chirplets and the chirplet transform), such that parts of objectsor media may be manipulated in phase space.

In one implementation, images can be edited such as in theirfour-dimensional phase space, through the manifoldizer.

The manifoldizer, as a form of interaction-design in one embodiment, canalso be used to manipulate action, to take action, or to edit action,thus not being limited to the principle of stationary action, but,rather, affording curatorial manipulation of action in spatializedaugmediated reality.

In this way devices like rockets, and the like, can be designed andmanipulated, amid a virtual environment that provides and createsreal-time analysis, Lagrangian (or Hamiltonian) modeling, and the like.Additionally, in one implementation, a part can be sculpted amid aninteractional environment in which real-time CFD (Computational FluidDynamics) is running while the part is being manipulated. For example,one can shape a rocket surface, while watching the “would-be” flow ofair currents and seeing the effects of real-time solutions to the NavierStokes equations, such that the shape is guided by more than mereaesthetics.

Thus, in some embodiments, the fluid flow itself can form part of theinteraction, e.g., a person can hold a part in a real wind tunnel, andsensors then sense the actual flow of fluid over the part and bring thisactual flow into the simulation.

FIG. 1F depicts examples of interacting with shared objects using anembodiment of the sensory and display apparatus 1300 of FIG. 1E.

FIG. 1F depicts one embodiment of the manifoldizer, showing a sharedmanifoldized interactional space and manifoldized display embedded in3-dimensional space, as well as a shared manifoldized interactor anddisplay embedded in another manifold (which may be a physical surfacesuch as a desktop, wall, or floor).

FIG. 1F illustrates an embodiment of the manifoldizer running with twowearable eyeglass devices (digital eye glass), the glass 180DEG and theglass 181DEG. The glass 180DEG and the glass 181DEG are extramissivespaceglasses (e.g., spatial imaging glasses). A DEG (digital eye glass),denoted as the glass 180DEG and the glass 181DEG, may include variouselements in various implementations, such as but not limited toorientation and an inertial sensor 180IMU, a LiDAR unit 180L, a visionsystem 180V, optical sensors, and display units 180D. LiDAR stands forLight Direction And Ranging.

The eyeglass device(s), or DEGs, of FIG. 1F may include a shade, to helpimprove eyesight, help increase contrast of view, or also to conceal theapparatus. In one implementation, single shade or two separate shades(one for each eye) may be used.

An example implementation of the shade as shown in FIG. 1F as a twoinches by 4.25 inches (approximately 51 -millimeters by 108 millimeters)standard-size welding shade. This allows the display 180D to be visiblein bright sunlight (e.g., using an ANSI Shade 5 to 7) or on a bright butcloudy day (e.g., using ANSI Shade 2 to 4), or for using the digital eyeglass for welding (e.g., using a darker shade), while seeing theelectric arc, such as by using HDR (High Dynamic Range) imaging.

The glass 180DEG can, in one implementation, include a singular glassthrough which both eyes see (like a welder's helmet). In oneimplementation, the glass 180DEG may further include a separate displayor mediation zone for each eye, within the one glass, or a separateglass for each eye, or a monocular glass for one eye.

The glass 180DEG may be supported on the head by a frame, such as mayinclude a head strap 180H and/or a mindstrap 180M. The mindstrap 180Mmay be split so that half of it goes below the occipital lobe and halfabove, thus making the eyeglass stay on during intense physical activitysuch as gymnastics, yoga, or swimming where water currents mightotherwise “brush” the digital eye glass off the wearer's face. A frontplanar surface of appropriately sealed instance of the glass 180DEG isoperational above or below water.

The digital eye glass 180DEG may be employed for a variety of usesand/or applications, such as but not limited to functioning as a seeingaid, vision aid, or the like, and may allow HDR vision, e.g., in whichthe wearer can see in relative darkness (or even in complete darkness,e.g., by way of a high dynamic range bolometer) while also looking intoan electric arc, or looking into bright car headlights in a dark alleyand still being able to clearly see the car's license number and thedriver's face.

In one implementation, removable shade 110 may be electrochromic and becontrolled by sensors, or the LiDAR unit 180L or a vision system 180V orby a combination of these. In this way, the glass 180DEG can adapt to awide range of vision conditions, such as change from indoors tooutdoors, sun, cloud, electric arc welding, bright lights, etc., asmight naturally occur, or even protection from deliberate or maliciouslighting attacks such as laser pointers or other sources of lights, towhich the digital eye glass 180DEG can afford protection.

In one implementation, sensors may be located around the periphery ofthe removable shade 110 of the glass 180DEG, and/or incorporateddirectly therein. Sensors may also, or additionally, be located aboveremovable shade 110, whether or not the removable shade 110 is present.When removable shade 110 is removed, this is considered as Shade 0,e.g., the setting of the removable shade 110 becomes Shade 0.

In various implementations, sensors above the removable shade 110 cantake the form of an infrared transmitter 180IRT, and an infraredreceiver 180IRR (detector or receiver). In one implementation, theinfrared transmitter 180IRT and the infrared receiver 180IRR worktogether as a range-sensing vision system, such as an infrared LiDAR orother type of 3D infrared camera system.

In one implementation, a vision sensor may be formed as a visible lightcamera denoted as visible light receiver 180VR. This allows for aphotographic image, perhaps a full color image, to be assembled togetherwith a range map or range image formed by infrared transmitter 180IRTand infrared receiver 180IRR. Likewise, a visible light transmitter180VT may be used to project onto subject matter in such a way as to bevisible by others, even those not wearing a digital eye glass.

In one implementation, a vision transmitter may operate in the visiblelight spectrum to illuminate subject matter such as surface 130 (e.g., atabletop) with visible content such as an advertisement, sponsor notice,counter, time/date, and/or other indicia such as a projection, which isvisible to wearers of DEG as well as to the naked eyes 198 of personsnot equipped with DEG.

In some implementations, the glass 180DEG affords two eyes a spatialimage, such as having horizontal parallax only, stereoscopic, and/or thelike. This may be done, in one implementation, using two separateinstances of the display 180D or a stereo display 180D in a single glassthrough which both eyes can see, or in separate glasses. Theinterpupillary distance (IPD) values from the 1988 Army Survey indicatean average of 64.7 millimeters for men, and an average of 62.3millimeters for women.

If we compute the average these two (e.g., for both genders), we get64.7+62.3=127.0 millimeters total, and dividing by two gives 127/2=63.5millimeters. This is equal to 63.5/25.4 inches which is 2.5 inches. Inone implementation, displays 180D are about 2.5 inches apart at theireye points in a mass-produced system, and may, in some implementations,provide for some degree of adjustment by the end user.

In some implementations, there may be a plurality of DEGs (digital eyeglasses) such as the glass 180DEG and the glass 181DEG, providing atwo-eyed, or monocular display capacity. In the case of a two-eyed view,there is provided an OD (Oculus Dexter) Point of Eye, 181POEOD, and anOS (Oculus Sinister) Point of Eye, 181PEOOS, e.g., “right EyeTap point”and “left EyeTap point”.

This capacity to capture true 3D images may be used in oneimplementation to create a lightspace collinearizer, which creates asynthetic EyeTap effect. In particular, in a true EyeTap system, rays ofeyeward bound light are diverted through an imaging system, processed,and re-rendered to be collinear with the original rays.

The synthetic collinearizer (lightspace collinearizer) works, in oneimplementation, by capturing a true 3D model and calculating the imagethat would have arisen by a camera if it were actually to have beenplaced inside the wearer's eye with the center of the lens (iris, nodalpoint, optical center, or the like) of the camera at the center of thelens of the eye.

Moreover, a 3D camera may be used in one implementation to synthesizethe effect of a spatial imaging device including a lightspace analysisglass, passing over the wearer's eyes, e.g., the entire visor of theSpaceglass can become effectively a lightspace analysis glass. This 3Dcamera thus generates the lightspace, lightfield, plenoptic, and/orholographic image of all rays of light passing through the Spaceglass. Aprocessor is therefore able to be responsive to an output of this(virtual) “lightspace analysis glass”, and the processor can furthercompute rays for a “lightspace synthesis glass” responsive to an outputof the processor.

In one implementation, the combination of 3D camera, processor, anddisplay effects a lightspace collinearizer.

Because of the true 3D nature in implementations of the vision system ofthe glass DEG180 and the glass DEG181, each instance of the glass isconfigurable to capture a true 3D image of the public subject matter 140such as a shared gaming table (e.g., a real or imagined or virtualchessboard, or the table surface 130 itself), and render this real view,or a computer-modified view of it, or any combination of these, as if itwere captured by a camera located at and/or substantially near thecenter of the lens of the Oculus Dexter (right eye), as well as anotherview as if it were captured by a camera located at and/or substantiallynear the center of the lens of the Oculus Sinister (left eye) of thewearer of the DEG (glass).

In one implementation, the glass 180DEG and the glass 181DEG can eachfunction as a Generation-5 glass. Moreover, the glass 180DEG can use itsown vision system plus information received from the glass 181DEG toconstruct an even more detailed, true and accurate 3D model of realitythan its own vision system alone. In this sense, a number of differentparticipants can share a computer-mediated reality in which the realworld is captured with high detail, precision, and accuracy.

In one implementation, two DEGs can cooperate, such as by time-divisionmultiplexing (e.g., alternately illuminating the scene to understand theworld), by code-division multiplexing (e.g., using different spreadingsequences or spread spectrum or spread spatialization patterns), bycollaborative sensing, and/or the like. In the latter case, the two ormore DEGs can be working together to illuminate and sense the publicsubject matter 140 and the surface 130. For example, while the infraredtransmitter 180IRT illuminates the scene, the infrared receiver 180IRRand the receiver 181IRR both sense the scene. The cross-sensing betweenthe glass 180DEG and the glass 181DEG provides additional sceneinformation through the extreme parallax that exists owing to the longerbaseline between the glass 180DEG and the glass 181DEG.

In one implementation, the result is a Synthetic Aperture LiDAR, or thelike, having a resolving power that may be mathematically broken downinto a separate small-parallax inter-glass baseline, and alarge-parallax intra-glass baseline.

In one implementation, the glass 180DEG may comprise or include or be awearable computer, or the wearable computer may be a separate unit, suchas that fits in a shirt pocket. In some covert embodiments of the glass180DEG, the apparatus may be totally self-contained, or it may betethered to a wearable computer for being placed in a pocket of a shirtof a wearer of the glass 180DEG. If it is tethered, the wiring may beconcealed, such as inside cloth pipes or cloth tubing, or the like,having the appearance of the Croakies™ eyeglass safety straps. In oneimplementation, the eyeglass safety straps have a tether clip 180T thatclips onto an article of clothing to provide strain relief of theeyeglass tether. The tether clip 180T can be a standard “crocodileclip”, an “alligator clip”, and/or a spring-loaded clamp, such as may beused for headsets, lavalier microphones, and/or the like.

In some implementations, the wearable computer may be placed on thefloor or table in an office or home space when the glass 180DEG is forbeing used in a seated position. In this case wearable computer might behoused inside the sensor-interface unit 120 (also called a tabletopunit), which includes a geophone 180G. The geophonic enabled instancesof the sensor-interface unit 120 (implemented as a tabletop unit) may beconfigured with sensors in it that sense and help with the process ofsensory input the wearable computer. In this case, sensor-interface unit120 can “listen” geophonically to sound vibrations in the table, such aswhen chess pieces are slid across the table or touched to the table, orwhen someone taps their finger on the table.

In one implementation, the LiDAR unit 180L and the LiDAR unit 181L,which sense the 3D object positions, and the like, provide data to thewearable computer along with sensor-interface unit 120, to providemultimodal sensory tactile feedback by way of geophone 180G and othersensors.

In one implementation, the geophone 180G in the sensor-interface unit120 can be a transmit and receive geophone, such as a unit made byClarke Synthesis which can even be bolted to the tabletop to providetransmission of a large amount of vibrotactile energy to the table thatcan be felt as a “click” or touch or taction by a user's fingers.

Alternatively, this force and/or vibrotactile energy can be usedsemi-destructively, such as for dramatic effect.

For example, in a game of “Augmediated Chess” the geophone 180G canshake the table and knock over real chess pieces when one player winsthe game, thus creating a dramatic visual effect even visible to thenaked eye 198 of a non glass-wearing participant.

Moreover, if multiple participants set their interface units down upon asurface 130, such as a tabletop, there can be multiple instances of thegeophone 180G and the geophone 181G in these respective units, such asto form an array of geophones. In one implementation, an additional IMU(Inertial Measurement Unit) in each of these units, and/or otheradditional sensors may be used to compute the relative position of theseunits, so as to form a phased array of sensors that can localizeacoustic or other disturbances in the surface 130.

In regard to human vision, as well as in regard to cameras, rays oflight are traced from their source, (e.g., a light bulb or other lightsource, or subject matter illuminated therefrom), to their destination,including through reflection, refraction, diffraction, and/or the like.This conceptualization of light may be referred to as the “intramissiontheory” of light. An alternative conceptualization may be referred to as“extramission theory”, such as with light as rays emanating from theeyes.

In one embodiment, “active vision” may be implemented through a specialseeing glass, e.g., the Generation-5 digital eye glass, wherein the eyecan behave as if it both senses and emits light.

In one embodiment, multiple digital eye glasses may share space, such asby using one or more multiplexers. One example of such a multiplexer isa TDM (Time Division Multiplexer) which is formed, in one embodiment, bytaking turns: one person transmits while the other person remains“quiet” (no light output). Then they switch places, taking turns totransmit so they don't interfere with each other. Such transmissions maybe energy-optimized in one implementation, so as to adapt more or lessenergy emissions to satisfy the need to see properly in an adaptivemanner. Adaptive active vision may be performed cooperatively, such asfor HDR (High Dynamic Range) by utilizing variously exposed gettings ofthe same subject matter, to save energy, as well as for the visionsystem to sense and/or see better.

In implementations of the fully bidirectional DEG that senses and emitslight in both directions may afford: extramissive active visionforward-looking; extramissive active vision inward-looking, such as tothe eye itself. This may include eye-tracking (e.g., as sensing) anddisplay (effecting), as well as spatial imaging of the environment(e.g., as sensing), and projection to the environment (e.g., aseffecting).

In this sense the glass 180DEG may send and receive light to and fromthe eye and may also send and receive light to and from subject matterin view of the eye.

In some embodiments, Spaceglasses, such as the spatial imaging glass180DEG and the spatial imaging glass 181DEG, sense both the environmentaround the user, and the user himself or herself. The spatial imagingcamera, such as LiDAR unit 180L, may in one implementation provide adepth map, such as to determine the spatial coordinates, positions,orientations, and the like, of the user's arms, hands, fingers, legs,feet, and/or the like, such as may depend on the visual orientation,field of view, and/or the like of the user. In one implementation, twokinds of scannable subject matter may be separately detected: (A)non-self subject matter (e.g., other objects in the room), and (B) self.

When the two meet, the system may track this and, and the situation maybe referred to, in some implementations, as “physical contact” or“taction”.

In one implementation, the computer 160 (a wearable computer or aWearComp unit) may contain a taction detector component that detectstaction (e.g., physical contact between the human body and something)and responds accordingly.

One example of a taction detector is the geophone 180G picking up thesound of taction, but the vision system can also observe this by 3Dscanning of a surface like a tabletop and determining when and if thehands or fingers touch it. In one implementation, this may beaccomplished by forming a point-cloud and noting when the point cloud ofhands or fingers intrudes upon the homography of the plane defined by agroup action, such as the example shown in Formula {1}.

In one implementation, a statistical significance test may be performedupon the degree of homographic intrusion to form a visual tactiondetector. Apparatuses, methods and systems that perform such measure ofthe degree of homographic intrusion may be referred to as a homographyintrusion estimator, or homographic intrusion estimator, and when suchan estimator is thresholded or used as a trigger for other action, sucha device may be referred to as a homography intrusion detector. In oneimplementation, detection and estimation theory, such as Neyman-Pearsontheory, or the like, may be applied to the creation of a homographyintrusion detector.

Some embodiments can be used stand-alone, or in combination with otherusers, collaboratively. Not all users need be wearing the DEG. Forexample, a shared computer-mediated reality exists among wearers of theglass 180DEG and the glass 181DEG.

This sharing can occur over a real or virtual manifold, such as anactual surface with the public subject matter 140. The public subjectmatter 140 can be on a piece of paper on a tabletop, chalkboard, or thelike, or it can exist entirely in virtual space as a manifold.

In one implementation, secret subject matter may be created in thevirtual world shared by participants wearing the glass 180DEG and theglass 181DEG. The user of the glass 180DEG may, in variousimplementations, choose to create, mark, annotate, alter, and/or thelike, the surface 130, only in virtual cyborgspace, with marking themanifold 146.

In one implementation, this is done by moving a body part 147 along atrajectory of the manifold 146, to create the manifold 146, including,in some implementations, in collaboration with another user of glass181DEG. The second user's body part 145 also edits and defines themanifold 146.

In FIG. 1F, the manifold is shown as one-dimensional manifold within atwo-dimensional instance of a surface 130, itself within athree-dimensional space. In one embodiment, a manifoldizer includes aprocessor (residing in the glass 180DEG or the glass 180DEG ordistributed there between), a sensor such as vision system 180V, and adisplay such as display 180D. In one implementation, the sensor and/orprocessor recognizes the position of body part such as the body part145, and tracks this part, and creates an integrated manifold path as acollection of point positions, such as may be stored in a database, andthen displays this path. The user sees, therefore, a time-integratedpath that continuously and/or periodically updates as the body part 145moves through or along space. The manifold 146 can be embedded in asurface like the surface 130, or it can be “floating in space” like themanifold 141. When it is embedded, as is the manifold 146 on a surfacelike the surface 130, it can be indexed or addressed by touching thesurface 130, and a surface contact sensor can comprise a planarhomography detection system that determines if the body part 145 is incontact with the surface 130. In this way, the processor can sensewhether or not the body part is in the plane of the manifold 146. Insome implementations, virtual objects can reside on the manifold 146.For example, a slider rendered as a bead-on-a-wire can appear and cantrack the body part 145 when it is in the plane of the surface 130, and“disengage” when not. This allows selective adjustment, interaction, orengagement, which may be referred to as multi-dimensionalmanifolidization (sliding up or down in a multidimensional space, forexample).

When the manifold is on a surface, it can be made visible to others evenwhen they are not wearing the digital eye glass (glass or DEG).

In one implementation, a person not wearing the glass can still see themarkings, if the participants decide to make them public. In this case,public markings can remain visible by extramissive visual projectionsfrom the glass 180DEG or the glass 181DEG, which markings compriseprojections, such as may be stabilized by Video Orbits stabilization.

In one implementation, the secret subject matter 340 may only be visiblewithin the database of collaborating users of the glass 180DEG and glass181DEG. Public subject matter 240 is visible to non-participants such asto the eyes 198 of a person not involved or not wearing DEG (digital eyeglass or glass).

The users of the data can decide whether to render the manifold 146 as ashared secret or public.

In various implementations, the surface 130 can be a desk, chalkboard,paper, flat, curved two-dimensional manifold, and/or the like.Alternatively, it can be a smart floor in which the surface 130 is afloor rather than a desk or tabletop. In one implementation, the floormay be configured with a pseudorandom texture upon it which may bedeliberately made from pseudorandom tiles, or naturally occurring by wayof varied texture, dirt, scuff marks, or the natural texture of woodgrain in a hardwood floor or the natural texture of stone or concrete ormarble or rubber or carpet. The texture forms a pattern. The homographyof that pattern, through algebraic projective geometry, forms a groupaction under a Video Orbits procedure running on the processorresponsive to an output of a 3D camera system formed by the infraredtransmitter 180IRT and the infrared receiver 180IRR. The group ofcoordinate transformations of the floor texture falls in an orbit givenby a formula similar to the example shown in Formula {1}.

Since the floor may likely remain in this orbit, as viewed by aplurality of wearers of DEG (digital eye glass or glass), then it can bedisambiguated from other matter not in the same plane as the floor.

In one implementation, the true 3D sensing nature puts the floor itselfin a true 3D model, and thus points on this 3D plane can be clusteredand segmented.

The secret subject matter 340 may comprise, for example, song lyricsdisplayed upon the surface 130 (also called a floor surface), visible toone or more participants in an andantephonic walk.

Aspects of the disclosed apparatuses, methods and systems may beconfigured for physical instruction (yoga, dancing, acting, martialarts, running, sports, etc.) by way of a virtual 3D instructor, orthrough multiple wearers of the spaceglasses. In the case of multiplespaceglasses, one's body movements can be scanned by another spaceglass,or by a spaceglass fixed in the environment, so that an instructor canassist in the collaboration.

Alternatively, the manifold may be a skipping rope or real physicalobject either laying on the floor like the manifold 146 or movingthrough free space like the manifold 141.

Alternatively, the manifold can be its own virtual object for which noreal-world counterpart exists. In this embodiment, the manifold 141 mayfloat in free space, as an image visible to users of the glass 180DEGand the glass 181DEG, but not visible to the naked eyes 198.

In one implementation, a user of glass 181DEG may use an extremity orbody part 143 to abakosculpt a 1D manifold in their shared 3D space.Abakosculpting, in some embodiments, may be performed with a light bulbor LED or other actual light source that draws long-exposure (“paintingwith lightvectors”) 3D photographs, as the trace 144. In otherembodiments abakosculpting may be performed with the fingertip as a“draw tool” to leave behind a drawn trace 144. Another user, such as thewearer (user) of glass 180DEG, can “reach out and touch” (using bodypart 142) the abakograph formed by the wearer (user) of glass 181DEG. Inthis case the abakograph may begin to glow a different color assigned tothe user of the glass 180DEG, while touched with the body part 142, orselected therefrom, so that the user of the glass 181DEG can see that heis sharing the abakograph with another collaborator.

The abakograph is represented as a 1D manifold in their shared 3D space,and either person can edit it, if the creator has given write permissionto this object.

In one implementation, a user such as wearer of the glass 181DEG canalso give read-only permission to the abakograph while also grantingwrite permission to spats that slide along the abakograph. In this waythe abakograph appears as a form of abacus to other users, who canmerely slide “beads” or other objects along it, but not otherwise alterit.

FIG. 1G depicts an example of interacting with shared objects along anabakographic trajectory in one embodiment using the sensory and displayapparatus 1300 of FIG. 1E.

FIG. 1G illustrates an abakographic manifold 180AM (hereafter referredto as the manifold 180AM) in one implementation. The manifold may bereal or virtual. For example, it can be an actual spiral or helix orother curved piece of wire held in the hand 180MD and the hand180MS of auser 180U, or it can be a totally virtual shared abakograph.Alternatively, it can be a 2D manifold or a 3D manifold, such as anautilus shell, a piece of paper, a map, or drawing, like a drawing orphotograph of a nautilus shell or spiral drawn on flat paper or on achalkboard or on the floor. It can even be a piece of modeling clay thatis held in the hands of the user 180U.

[000275] The right hand 180MD (“manus dexter”) of the user 180U touchesthe manifold 180AM (which may have a spiral shape) and the glass 180DEGsenses this, in one implementation, through its active vision system byway of sending light to the hand 180MD and receiving the light back,thus determining whether or not the hand 180MD has made contact with themanifold 180AM.

An object 180O (a real object or a virtual object) is present that canalso be touched. Here it is presented as a “bead” on a wire, and whenthe hand contacts its surface, the bead is “highlighted”—rendered in adifferent color, corresponding to user 180U—let's say, red for the user180U and blue for the user 181U.

In one implementation, the bead is a spat, meaning that it can representan environment map when and if desired. Here it represents a panoramicphotographic virtual environment as captured from a center-of-projectionthat aligns with the center of the sphere formed by object the 180O.Here the object is spherical, but it could also be tree-shaped, orclown-nose shaped, labrador-shaped, giraffe-shaped, or anything else.

In some embodiments an object is actually spherical in reality butpresented to the viewer as another shape such as a polyhedra or gameitem, or a photographic spat showing the texture map as if rays of roomward bound light were passed through it and recorded thereupon.

In one implementation, touching an object like the object 1800 with thehand 180MD while at the same time touching another object 180P with thehand 180MS associates the two objects with one-another, highlightingboth of them, as well as the path there between, in red.

Subsequently when user 181U touches the manifold, his instance of theobject 180Q begins to glow blue in both his instance of the glass 181DEGand the first user's instance of the glass 180DEG, and also the pathbetween the object 180O and the object 180Q begins to glow magenta (asummational mixture of red and blue).

These paths, defined along the manifold 180AM, are paths along which theselected objects can be slid in virtual shared space.

Various different kinds of gestures may be employed. These include:touching the manifold 180AM; touching a spat such as the obj ect 180O;grasping the spat such as the object 180Q, between thumb and indexfinger of the hand 181MD; and touching while punching a spat, with indexfinger of the hand 181MS, while also pushing the knuckles of the hand181MS against the spat.

In one implementation, a manifoldization gesture sensor recognizes thedifference between a gesture that slides along the manifold and thentouches an object, and one that just touches the object afterapproaching through three-dimensional space but not along the manifold180AM.

In one implementation, objects like the object 1800 may be representedas “fuzzy” clouds that have “snap” properties, allowing near touches tohighlight them. These clouds may be displayed around the spats as anaura, for example, which itself gradually and fluidly highlights onapproach, thus allowing analog selection, and degrees of selection, suchas using actional analytics (e.g., the time-integral of inversedisplacement, or presement, where presement is reciprocal absementinside the time integral, in which absement being the time-integral ofdisplacement).

In one implementation, selection is performed by an actional integratorthat integrates reciprocal displacement, to determine the strength ofthe selection aura.

Additionally, an exposure integrator can be used in some implementationsto time-integrate exposures or exposers such as hand-gestures used tocreate selectioners.

In one implementation, a “crush” gesture is one that indicatesdestruction of an object such as the object 181BB which can “die” like aburst bubble.

In some embodiments, the disclosed apparatuses, methods and systemsinclude the Pi-menu, the Tau-menu, and the Spat-menu.

In one implementation, the Pi-menu is selected by approaching an objectfrom below, in which case the lower half of the object is given theselection aura. A Tau-menu may be selected from multiple directions ofapproach, such as by grasping (e.g., thumb and index finger). A Spatmenu may be selected by crushing into a sousface. A sousface is, in oneimplementation, the opposite of a surface (French etymology from“soffit”). A touch surface may be created by touching an object, but areach-into-the-object gesture may be distinguishable from touching andmay be effectuated in one implementation by reaching through the objectand connecting with its interior. Here we see the spat menu 181SM appearbecause the hand 181MS reaching into the object, passed through itscenter, and then came at the inside of the object to touch its sousface.

Once a sousface touch is detected (e.g., a touch that goes through theobject, and for example, “pops” the bubble, and grabs onto its “soapy”fragments from the other side), a 3D clock face menu appears in oneimplementation. This goes beyond a 3D pie menu, because also the spat isused as a framework for 3D selection beyond the decorative. The hands ofthe clock can also be manipulated separately, in addition to a full 3Dspat thereupon. The finger then is tracked in its manner of departure(e.g., direction, velocity, acceleration, jerk, jounce, and/or the like,as well as displacement, absement, absity, abseleration, and/or thelike), and the spat is expanded (“pulled apart”) to enlarge the menu.Here the index finger of hand 181MS pulls away from the center of theobject to select the four-o-clock position as the menu selection 181M,here indicating a moratorium on the process. A four-o-clock gesture isencoded as an end gesture, for an application that is close worthy.

It should be noted that the above is provided merely as one illustrativeexample and is in no way meant to limit scope. For example, in oneimplementation, a 3D direction of entry and/or departure of one or morebody parts of one or more users in a shared augmediated realityenvironment may be employed. In various implementations, each of thesegestures can be metagestures, autogestures, or the like. By way ofexample, a user may punch the bubble, poke it with one finger, or touchit with two fingers gently to rotate it as a select-and-rotate gesture,or touch it with the two middle fingers while clenching the index fingerand thumb simultaneously, squeezing the index finger and thumb harder orsofter to establish a continuous multi-touch autogesture that iscontinuously variable and evolves dynamically while the gesture istaking place. In some implementations, such dynamic “fluid gestures” canoperate continuously and give rise to a continuous rather than discrete“trigger-based” user interface modality. Examples can include an RWM(Reality Window Manager) interface in which windows are touched, rubbed,moved, burst, broken, fixed, repaired, “washed” (with a rubbing motionto erase the text from a text window for example), and the like, allwith multi-touch metagestures, multi-touch autogestures, and the like.

The spat menu 181SM is not a mere clock face but is three-dimensional.For example, if the hand pulls straight away toward the user, or awayfrom the user, other actions may be taken.

Embodiments of the disclosed apparatuses, methods and systems may beconfigured to work in any number of dimensions. Here we see anembodiment employing an abakographic manifold 180AM, but embodiments mayalso be configured to employ, simulate, and/or manipulate the 3D objectslike modeling clay. The user 180U holds clay that is sculpted by thehand 180MD and the user is guided by overlays to make a perfectsculpture in real clay. In one implementation, this may be effectuatedvia a wearable 3D holographic vision where the clay itself can be anexposer to long-exposure sculpting effects, beyond mere augmediatedsculpting. In one implementation, the user may then insert this(virtually) into a 3D printer to be printed.

The clay is not necessary as the action can be performed in open-air,but the clay does provide useful tactile feedback that improves thetoposculpting experience and accuracy.

Other embodiments include special shoes with special markers that allowthe user 180U and the user 181U to do toposculpting through actions likewalking, dancing, or interacting. They can, for example, dance togetherand generate abakographs with their feet, as an art form, or as apractical way of measuring physical fitness and kinesiological data.Embodiments may be configured to work with dance shoes, yoga shows,mindfulness exercises, aerobic exercise, and bicycling where theabakographs can be used to measure performance and generate performancemetrics.

Moreover, in one implementation, a comparametric interaction amongmultiple users can be used to generate an abakogram or other datastructure from which an abakograph can be generated, such as at a latertime.

In one implementation, Toposculpting may also be performed in or withfluid, using the fluid as tactile media. For example, compressed orpressurized fluid can be used as a fluid flow field against which tocreate tactile sculptures.

FIG. 1H depicts an example of a method for interacting with sharedobjects using the sensor and display apparatus 1300 of FIG. 1E.

FIG. 1H shows a flowchart depicting aspects of the disclosedapparatuses, methods and systems in one embodiment.

FIG. 1H is a flowchart depicting some aspects of the disclosedapparatuses, methods and systems. A topological sensor 1C10 determinesif an object is in range of the hand 180MD of the user 180U. If not, Itwaits until the hand is in range of the object. Once the hand “intrudes”sufficiently into the “space” of an object such as the object 1800, amanifoldization display, begins to play back a recorded or live image ofa situation such as live video from the spat so selected.

In this scenario consider a veillance system that is neither watchingfrom above (surveillance, e.g., as police watch prisoners) or watchingfrom below (sousveillance, e.g., as citizens photograph police).Veillance is the act of watching.

The veillance system works within an Internet-of-Things (IoT) orSMARTWORLD framework. Examples include streetlights that have a camerain each light to automate the adjustment of light output. A citizenwalking down the street can edit the space (assuming everyone haswritten permission, so they can request light when needed or, so theycan suggest changes) and interact to some degree, with some amount ofwrite permission.

The citizen interacts with a map of the city to select a spat. Once herfinger touches the spat, the spat begins to play live video. We callthis a maniplayer (TRADEMARK) apparatus, a manifold (dis)player thatplays back live video on a globe as if the globe were the very fisheyelens that recorded it. Other projections can include for example amercator projection on a cylinder.

Consider the sphere of the object 1800 as the maniplayer surface, wherethe maniplay 1C20 beings to play live video from a streetlight cameraonce the camera has been selected. The user 180U can see herself on thevideo and locate a problem where the branches of a tree are blowing inthe wind and causing the streetlights to come on at full brightness whennobody is on the street, which is wasting a lot of electricity. Shecaptures the live video of the situation and forwards it to a wiki, soshe can play it back next time she is attending a City Council meetingto have the problem fixed. Meanwhile she uses the spat menu 181SM todraw a veillance mask with her finger to cover up the tree branches sothat they can be ignored. She posts this proposed mask on the wiki.

Such citizen involvement in veillance turns what might otherwise be asurveillance society into a veillance society.

Some embodiments of the disclosed apparatuses, methods and systems mayfacilitate interaction design with hand gestures at or with real orvirtual manifolds—e.g., to shape them—as a way of interacting.

Manifolds are locally planar: the “world is flat” (approximately) if youlook at a small enough patch of it. In one implementation, a Lie algebramay be employed to consider the behavior of a topological manifold inthe neighborhood of one point such as the identity. Thus, thetopological sensor 1C10 is implemented as a local planar comparison atthe neighborhood of one point-of-contact with the object 1800 regardlessof shape (e.g., sphere).

In one implementation, once a body part like the fingertips of the hand180MD intersects an object like the object 1800, a sustainment detector1C30 determines if the touch was an accidental brush or a sustainedtouch. If accidental, the maniplay may be terminated by the maniplayterminator 1C40. In one implementation, sustainment detection may bebased on a duration, sequence, pattern, degree, and/or the like oftrusion and/or taction.

Otherwise maniplay continues and a gesture sensor 1050 senses fromvarious possible gestures at, with, in, or on the object 1800.

A slide gesture results in the ability to “slide” the spat along a path,for example, to select the next streetlight along the street, or tointerpolate camera views between streetlights.

This provides a real-time interactive street view in which the user 180Ucan see that there is a man with a gun hiding behind a bush furtheralong her route. She slides the spat of the object 1800 further along tosee better, and then selects the spat between her thumb and index fingerand drags it to a Police Box in the lower right hand corner of hervirtual page. The gunman is taken into custody while she by then hasalready begun to take an alternative route to avoid the gunman, meetupwith a police cruiser to file a report, and then walk home along a routethat is not longer than her original route, owing to the planningfunction of an embodiment of the disclosed apparatuses, methods andsystems, which guides her home efficiently.

Subsequently she provides a close gesture by pulling a clock face tofour-o-clock after she has saved the recorded video into the Police Box.

On another occasion she observes a squirrel running along a wire, andsends a cute picture to her weblog, and is able to perform variousgestures with the “camera beads” along the street. In oneimplementation, a touch gesture highlights a spat, and ends to lowlightit when the touch is done. A squeeze gesture selects a spat, which canbe crushed with the knuckles, e.g., to close it more permanently,especially true of an app that has excessive advertising, and is trulyclose worthy.

In this case the spat is deleted from her list, so she doesn'taccidentally open it again unless she wants it again and puts it back inher list of cameras. In this way, a “shortlist” of favorite or mostuseful cameras is retained in one implementation, whereas the lessuseful ones can be moved off the shortlist by the object deleter 1060.

In one implementation, when a spat is squeezed, e.g., between the thumband index finger, a special menu may be generated which allows not onlythe spat itself to be moved like a bead on a wire, but also the whole“wire” itself, e.g., the user can “pull” over to an adjacent street (andsee through a new row of cameras on the next street).

A “pull” on spat menu 1C70 may indicate to the system that achange-of-path is requested, not merely a change in position along onepath.

This moves a sub-manifold, such as a street line, over to a next street,in-between (e.g., by synthesizing a view of a back alley where nocameras are present, using interpolated views from streets on eitherside of the alley), and/or the like.

A stronger gesture is a “pulls” gesture which, in one implementation, isa plurality of pull gestures, or is done by more than one finger to“pull” at the “wire”.

In one implementation, the pulls gesture is indicating a desire to moveup one dimension, and deal with the 2D manifold.

Thus we have a hierarchy of gestures in one embodiment: (A) a OD (zerodimensional) gesture: touch or move spats along a “wire” (e.g., movealong a street); (B) a 1D (one dimensional) gesture: move the “wires”themselves (e.g., move across from one street to another); (C) a 2D (twodimensional)gesture: move the manifold of manifolds: e.g., the mesh ofthe city, or the like; and (D) a 3D (three dimensional) gesture: movethe manifold of the manifolds of the manifolds, e.g., move the embedding3D space itself.

The submanifold mover 1C80 moves, in one implementation, at the streetlevel, whereas the manifold mover 1C90 moves at the map or chart level.

Chart-level mappings and manifold manipulations may be referred to, insome implementations, as the planifolds™ apparatus, which may bedisplayed in one implementation like a PPI (Plan-Position-Indicator) ofa radar set, with the user in the center, and the city shown radiallyoutwards from the user-centered design perspective.

Planifolds can be grasped and moved like the other manifolds.

Planifolds can be displayed on any piece of scrap paper with any textureon it and be folded and manipulated.

In some implementations, the disclosed apparatuses, methods and systemsmay be configured for making new parts, using the hands to sculpt in 3D,make objects, modify objects, and the like, in virtual space oraugmediated space, e.g., using clay or scraps of paper or any sort ofbricolage for rapid tactile prototyping.

In another embodiment, doctors use toposculpting to assist patients. Apatient with a wound can be assisted by way of a 3D printer that printsskin or skin like material. The physician uses toposculpting to shapethe skin printing to the patient's body.

FIG. 2A depicts another schematic example of the sensing and displayapparatus 1300 of FIG. 1E.

More specifically, FIG. 2A illustrates principles of metaphor-freecomputing and natural user interfaces as used in some embodiments withthe sensing and display apparatus 1300 of FIG. 1E. There is depicted ashared computer-mediated reality, such as the first augmediated-realityspace 1000 of FIG. 1A, among wearers of the digital eye glass 180 andthe digital eye glass 190. The surface 130 is a piece of paper on atabletop, on which the public subject matter 140 includes printedmatter. A participant (user) wearing the digital eye glass 180 may, inone implementation, employ a drawing tool 201, which may be called adrawtool™ system. The drawing tool 201 may include an inertialmeasurement unit 204 and/or the like. The drawing tool 201 may include atool point 200 configured to be a virtual device or may be configured toinclude a physical marking tool (having ink or graphite, or othermarking material). A geophonic sensor 203 is configured to pick up thesound or other disturbances associated with the drawing tool 201; inthis way, the drawing tool 201 is configured to interact responsively,in association with other sensors in the drawing tool 201 that sensecontact with the surface 130. The virtual subject matter 202 may remainsecret (e.g., not viewable by people not using an instance of thesensing and display apparatus 1300). The virtual subject matter 202 maybe created, displayed, presented, and/or the like in the virtualenvironment in the first augmediated-reality space 1000, and is sharedby the participants wearing the digital eye glass 180 and the digitaleye glass 190 (in the first augmediated-reality space 1000). The user ofthe digital eye glass 180 may choose to mark the surface 130 only infirst augmediated-reality space 1000 (e.g., cyborg space), or to markthe surface 130 with the physical markings. A person not wearing anyinstance of the digital eye glass may still see the virtual markings forthe situation where the participants (users using their instance of thesensing and display apparatus 1300) decide to make the virtual markingsavailable in a public manner. For this case, the public markings, suchas the public subject matter 240, may remain visible, e.g., byextramissive visual projections from the digital eye glass 180 or thedigital eye glass 190, which markings include projections, such as maybe stabilized by a video-orbits stabilization program 403 depicted inFIG. 1AA. Alternatively, markings may remain visible by way of themarking material such as graphite or ink.

FIG. 2B depicts Video Orbits stabilization and comparametric alignment,and the like in one embodiment.

FIG. 2B depicts the Video-Orbits stabilization system in one embodiment.Video orbits may work in a variety of ways in various implementations;e.g., pixel orbits and/or voxel orbits. In one implementation, pixelorbits may employ the homography of one or more rigid planar patchesand/or segment(s) thereof. In one implementation, voxel orbits track the3D positions of objects, identify planes, and/or track associated pointclouds.

In one implementation, pixel orbits may employ 2D cameras while voxelorbits may employ 3D cameras. In one implementation, pixel orbits andvoxel orbits may be combined, such as where apparatus 1300 includes bothone or more 2D cameras and one or more 3D cameras. For example, whereapparatus 1300 includes one 3D camera and two 2D cameras, orbits may betracked, e.g., using 2D vision, stereovision, true 3D vision, and/or thelike and/or may combine these results, such as using a confluenceaggregator, certainty combiner, and/or the like, where various estimates(e.g., how a surface texture is manifested in pixels, how a shape ismanifested in voxels) may be combined, e.g., through a weighted sum. Inone implementation, weights may be certainties as determined, e.g.,comparametrically (e.g., via comparametric equations),superposimetricall (e.g., via superposimetric equations), or by anyother error estimate. Thus, in one implementation, albedo (lightness,color, texture, etc., of a surface or object or the like), range, and/orthe like may be combined, e.g., in an optimal and/or optimized manner.To the extent that albedo and range both give the same motion, aconfluence combiner may, in one implementation, be used on the estimate.Sometimes a surface or object will be of uniform albedo with no patternssuch that 3D range information may be more accurate. In someimplementations and/or uses, such as tracking egomotion against thenight stars or sky or distant objects, range measurements are lessreliable and the 2D Video Orbits may be employed. In someimplementations, 2D Video Orbits and 3D Video Orbits may be combined,making good use of both albedo and range information, for trackingobjects such as rigid planer patches.

Implementation of 2D (pixel) orbits, an orbit can be considered as a setof transformations that can occur under a group action.

Consider the comparametric group action given by g(q)=f(kq), defined bythe following formula:

g(f(q(x))=f(kq((Ax+b)/(cx+d))),

where f is the domain of the transformation, and g is the range of thetransformation. Notably, f and g are both domains of aphotoquantigraphic response function, of which the domain is thephotoquantity, q (quantimetric units).

Moreover, q is the range of a mapping from x, where x is a two by onecolumn vector of spatial image coordinates, e.g., x=[x1; x2], itstranspose, and/or as x=[x1; x2], using the boldface x.

The matrix A captures scale in x1 and x2, as well as shear in x1 and x2.Together these can also define rotations in x, which are captured whenthe 2×2 matrix A is a rotation matrix.

The row vector c is of dimension 1×2 and, in one implementation,represents chirp, or chirprate, in x1 and x2 (it has two degrees offreedom, which can be alternately thought of as chirpiness and directionof chirp).

The constant d in the above equation is a scalar constant.

The mappings g form a group, and the group has five real parameters: A,b, c, d, and k, which may, in one implementation, be arranged in amatrix, [A, b, 0; c, d, 0; 0 0 k].

In an implementation, there are only four degrees of freedom (9 scalardegrees of freedom) because we can divide all elements of the matrix byd, whenever d is not zero (or whenever d is not close to zero).

Subject matter in view of a spatial imaging glass, which is also knownas the Spaceglass™ apparatus, e.g., subject matter in view of a sensingand display apparatus, often falls on a flat or somewhat flat, or planaror somewhat planar surface or entity.

When subject matter is sensed, especially in urban or indoor settings,there are often a large number of surfaces that are somewhat flat. Manyobjects like buildings and furniture are made of somewhat flat surfaces,which may be automatically segmented by the Spaceglass apparatus.Looking, for example, at a window, we can see there is content on aplanar surface, but also there may be content behind and/or in front ofthe window. Consider, for example, a window made of separate panes ofglass, or, alternatively, a fence, mesh, screen, jail door, grille,and/or the like, which may have some planar content but also may haveobjects in front of and/or behind it.

Subject matter, such as the subject matter 2B10, may be viewed fromdifferent vantage points, as the subject matter 2B20, and the subjectmatter 2B30. Some of the subject matter 2B10, the subject matter 2B20,and the subject matter 2B30, respectively, is in the plane defined bythe subject matter 2B16 (such as, a plane), the subject matter 2B26(such as, a plane), and the subject matter 2B36 (such as, a plane), butother of the subject matter, respectively the subject matter 2B 15, thesubject matter 2B25, and the subject matter 2B35, is out of the planedefined, respectively, by the subject matter 2B16, the subject matter2B26, and the subject matter 2B36.

Each of these three “gettings” of the same subject matter, arerepresented as the getting 2B1, the getting 2B2, and the getting 2B3. Insome implementations, a getting can be a picture, a set of measurementsfrom an active vision system, such as the measurement to a response,and/or the like. In this sense, a getting can be a comparametricmeasurement, a picture, video, quantigraph, or the like, or it can alsobe a superposimetric measurement, or collection thereof, e.g., made inresponse to a stimulus, or collection of stimuli, such as one or moreexposures to electromagnetic radiation.

For example, we can “dust” a tabletop (e.g., abakographically), byfiring an electronic flash source at the table while measuring (e.g.,capturing, photographing, and/or the like) the response.

In one implementation, the flash is configured as an aremac, such as astructured source of illumination.

An aremac (the etymology of which is “camera” spelled backwards) may actas a source of illumination, and together, the camera and aremac(“aremacamera” or “macam” for short) may, in one implementation, form asuperposimetric measurement and response (excitation and sensing)system.

The aremac can provide varying amounts and patterns of illumination forvarying exposures.

A sensor, such as a camera, may be configured with varying gain, as mayits source of stimulus, such as its illumination, if present. Thus, acamera or macam can provide variously exposed gettings of the samesubject matter, such as the getting 2B1, the getting 2B2, and thegetting 2B3.

The getting 2B1, the getting 2B2, and the getting 2B3 representdifferent transformations on the true subject matter 2B. The getting 2B1 was “gotten” (e.g., received) at a low level of exposure orillumination with the subject toward the left side of the field-of-viewof the sensor. The getting 2B2 was “gotten” with a moderate (medium)amount of gain or sensitivity or exposure or illumination, with thesubject matter approximately in the middle of a field of view, butlooking upward a bit. The getting 2B3 was “gotten” with a high amount oflight, in other words, “overexposed”, with the subject matter toward theright side of a field of view.

These three gettings represent three different interpretations of thereality in the subject matter 2B.

They may be related to one-another, in one implementation, by thetransformation of the form g(f(q(x))=f(kq((Ax+b)/(cx+d))), e.g.,comparametrically, projectively, superposimetrically (e.g., wheredifferent illuminations were used from one to the other), and/or thelike.

In this case one getting may be picked, let's say the middle one, thegetting 2B2, and the others may be expressed in terms of the getting2B2. There will be one transformation, given by A12, b12, c12, d12, andk12, that will relate the getting 2B1 to the getting 2B2, and anothertransformation, given by A23, b23, c23, d23, k23, that will relate thegetting 2B2 to the getting 2B3.

The “truth” about what is happening in a scene may, in oneimplementation, be reconstructed by constructing estimated compositegetting 2B0 from the getting 2B1, the getting 2B2, and the getting 2B3.The getting 2B0 represents everything the Spacelgass “knows” about thesubject matter 2B.

In this sense, a de-chirped representation of the subject matter 2B06may be provided that falls in a plane such that points in the plane areuniformly spaced.

In one implementation, a chirp is like the sound a bird makes, e.g., anote or tone or periodic waveform where the frequency increases ordecreases. In the case of vision, a chirp is a something that goes up ordown in frequency as we move across it. Subject matter of the getting2B1, for example, is “up-chirped” in the sense that the subject matter2B16 (planar content matter) has spatial frequencies that start out low(e.g., the window muntin or grille) and increase in spatial frequency aswe move from left to right (e.g., in increasing dimensions along x).

In one implementation, the subject matter 2B36 of the getting 2B3 is“down-chirped”, e.g., it starts out at a high pitch, and goes down to alower pitch as we move from left to right in increasing x.

In one implementation, the subject matter 2B06 of the getting 2B0 (thecomposite getting) is de-chirped, so that units of the plane areuniformly spaced in an internally stored array in the Spaceglass system.

In one implementation, each plane from reality may be stored into ade-chirped quantigraph q(x,y) in the Spaceglass, such as by segmentingplanes from reality and applying transformations such as indicated bythe corner 2B11, the corner 2B12, the corner 2B13, and the corner 2B14that map directly from the getting 2B1 to the getting 2B0 rather thanfirst to one of the gettings like the getting 2B2.

This may be achieved, in one implementation, by applying transformationsof the form A1 b1 c2, d1 k1 to the getting 2B1 toward the getting 2B0,and of the form A2, b2, c2, d2, k2, to the getting 2B2 toward thegetting 2B0, and of the form A3, b3, c3, d3, k3, to the getting 2B3toward the getting 2B0, each of these three transformations providing anestimate of the getting 2B0. These three estimates are then combinedtogether as a combined the getting 2B0, according to the certainty rulesof comparametric image compositing.

The subject matter 2B16, the subject matter 2B26, and the subject matter2B36 will align in confluence to the subject matter 2B06, whereas thesubject matter 2B15, the subject matter 2B25, and the subject matter2B35 will misalign in fewer confluences to the subject matter 2B05,because this subject matter is off-plane.

Likewise, clouds in the background seen through the window willmisalign.

The result is a confluence composite of the mullion and muntins of thewindow and the window frame, and the glass (e.g., dust on the glass,texture, pattern, and/or the like, of the plane).

When we look at subject matter with a window in the field of view, wesee the subject matter of the foreground, e.g., darkly, in silhouette,and/or the like. As we look to one side, when the window goes to oneside of our view, e.g., when looking down toward the floor, the AGC(Automatic Gain Control) of some cameras will “gain up” rendering somebut not much detail in the face of a person standing in front of awindow, for example, as the subject matter 2B25. If we look down somemore, or off to one side, with the center of our camera pointed atsomething in the room that is dark, such as a black cloth or darkopening of a doorway into a closet where some black clothes are hanging,with no light turned on in the house, the AGC will “gain up” and renderthe person's face of the subject matter 2B35 very clearly, but theclouds and sky outside will be “washed out” white (e.g., overexposed).Even the mullion and muntins will “whiteout” to some degree.

Thus, contributing to the getting 2B0, most of the bright areas of thesubject matter like the sky and sun, come from the getting 2B 1, andmost of the dark areas like the almost silhouetted face of the subjectmatter 2B35 come from the getting 2B3. The getting 2B2 will provide muchof the mid tones.

This may be achieved, in one implementation, by comparametric imagecompositing over a manifold defined by the 3D shape of the subjectmatters. The simplest shape is the plane, and that can be done veryaccurately, but the face is more complex, being aggregated of lesseraccuracy.

Thus, the getting 2B0 can be used to estimate the true subject matter 2B(as an estimate), and the subject matter 2B00, of the reality of thesubject matter 2B.

Since the plane can be “gotten” at great accuracy, this may serve as ahomographic user-interface in the Spaceglass. Likewise, with othergeometric shapes like spheres.

Subject matter gettings may be combined, e.g., by confluence, such asusing the Certainty Function of Comparametric Equations theory.Alternatively, a confluence combiner may be used.

In one implementation, confluence is the degree to which entities cometogether or should come together in agreement. If there is agreement,e.g., if two gettings agree on reality, with confidence, they getcombined more certainly. This may include modifications to the theory ofComparametric Image Compositing.

Confluence Compositing

Suppose we wish to conflate two signals “a” and “b”. Signals may beweighted by certainties, by confluence, and/or the like. In digitalimplementations, confluence can be implemented by an exclusive OR gate,e.g., (a+b) (a XOR b) to weight more strongly agreement.

In an analog implementation, consider the product [ab]. The product [a]times [b] is large when [a] and [b] are large. When one of them isnegative and the other is positive the product is negative.

Therefore, confluence could be expressed, in one implementation, as:

c=(a+b)ab.

In an alternative implementation, confluence may use the following:

c=(a+b)exp(ab).

Such confluence compositing may be used, in some implementations, forconfluence sensor 402, for combining an audio trusion detection with avisual trusion detection, for other forms of multimodal sensory fusion,and/or the like.

In one implementation, to save power, the illumination from theSpaceglass may be adjusted, such as in a manner that is adaptive tovarious subject matter. For example, subjects that are nearby may beilluminated with less light, whereas subjects that are further may beilluminated with more light, e.g., for (1) best exposure; (2) to savepower, energy, and action; and/or the like.

In some implementations, action may include Hamiltonian Action (e.g.,the time integral of kinetic energy, T, plus potential energy, V),Lagrangian Action, the time integral of T minus V), and/or the like.

In some implementations, the Principle of Stationary Action may beapplied to an orbit of the projective group of coordinatetransformations.

An adaptive PEAMS (Power, Energy Action Management System) may, in oneimplementation, may include: (A) a passive getting of low sensitivity isperformed; (B) an analysis is made of the getting, to determine acertainty of the getting: for a getting that is an image exposure, thecertainty is a certainty image; (C) If there are areas that areuncertain (e.g., based on comparison with a threshold, standard, and/orthe like), a getting of greater sensitivity is performed; and (D) Theprocess is repeated until sufficient exposure sensitivity is attained,such as based on comparison with a threshold, standard, and/or the like.

In some situations, a SPEAMS (Superposimetric Power Energy ActionManagement System) may be employed. The SPEAMS system, in oneimplementation, may include: (A) an active getting of low sensitivityand low output is performed; (B) an analysis is made of the getting, todetermine a certainty of the getting; for a getting that is an imageexposure, the certainty is a certainty image; (C) if there are areasthat are uncertain, an active getting of greater sensitivity isperformed; (D) the process is repeated until sufficient exposuresensitivity is attained; (E) these results are conflatedcomparametrically; (F) the resulting confluence image is logged as alightvector to a getting of low output; (G) the output is increased andthe process of multiple gettings (e.g., exposures) is repeated tocapture the lightvector resulting from an emission of medium output; (H)each of these lightvectors manifests a superposimetric image space, eachhaving a particular energy cost, i.e. for each lightvector there aremultiple gettings that each require a burst of light output of aparticular strength (e.g., low output, medium output, etc.); (I) anenergy cost function may be computed for each burst of light produced,and these energy cost functions are accumulated as an actional cost,associated with the computation of a superposimetric lightspace; and (J)the process continues until the subject matter or scene is lightspacedto a sufficient degree, e.g., until an error term in a lightspace costfunction is minimized sufficiently.

In this manner, action may be substantially conserved.

In some implementations, lightspace can also be computed using theprinciple of stationary action (Lagrangian) or to conserve Hamiltonianaction (total action).

In one implementation, as with a PID (Proportional, Integral,Derivative) controller, we may determine and/or estimate motion in allof its integrals or derivatives. For example, the orbit of the camera orhead or eyeglass is estimated in displacement, in a feedback loop, overderivatives and integrals of displacement, motion through space, and/orthe like.

In one implementation, this may be achieved via an Abseleration, Absity,Absement, Displacement, Velocity, Acceleration, Jerk, Jounce, etc.signature computed from the motion, estimated motion, parameters, and/orthe like.

FIG. 2C depicts an absement-based signal processor in one embodiment.

With reference to the example illustrated in one embodiment in FIG. 2C,a Set Point, denoted “SP” is established as the signal 2C70. This maybebe, for example, an abakographic gesture input by way of a body part, ora drawtool such as the drawing tool 201 as shown, for example, in FIG.2A. The signal 2C70 may, alternatively, be input by way of a lightsource held by the user, such as an LED (Light Emitting Diode) light, oran array of LED lights, or a special glove with lights in it, or by alight stick supplied with the spatial imaging glass™ apparatus. In thisway, the Set Point can be a point in space, which is then drawn or movedthrough space with a particular trajectory, and particular kinematics,sensed, observed, and in reality.

The Principle of Stationary Action may, in one implementation, becomputed on the kinematics observed, as compared with the actualkinematics, and thus there is manifested a control system, effectorysystem, and/or the like, which senses and tracks trajectory andmovement, e.g., for general-purpose gesture sensing, abakography,abakographic user-interfaces, toposculpting, and/or the like.

The set point input produces a time-varying waveform signal r(t) whichmay be an audio signal, a visual signal, or a combination of these, andmay also have other components such as olfaction (chemical sensors)and/or the like.

Process 2C10 may, in one implementation, capture aspects of the world,such as objects and their motions, as they actually exist in reality.This reality may be modeled, for example, by way of various instances ofthe processor 2C35 denoted, mathematically, as K_{−m}, . . . K_{−3},K_{−2}, K_{−1}, K0, K1, K2, . . . K_n.

A “Process Variable”, PV, may be captured (e.g., sensed) from realityand manifested as process variable 2C15. A sum of the Set Point and thenegative of the Process Variable, e.g., [SP]-[PV], is computed by theadder 2C20, to provide the error signal 2C25.

[000395] The error signal 2C25 is denoted, mathematically, as e(t). Inone implementation, the error is an error in whatever units are measuredby the Process Variable and the Set Point. For example, this error maybe an error in position, displacement, distance, or the like, as sensedby displacement sensors, position sensors, distance sensors, or thelike, such as the LiDAR (Light Direction And Ranging) apparatus, such asa Kinect™ depth sensor, a Primesense™ depth sensor, and/or a SoftKinetic™ depth sensor, which purports to measure “depth” (distance fromsensor or displacement, for example).

Control systems may use the sensed quantity, as well as one or both ofits time derivative and time integral. Kinematic processors 2C30 computethe kinematics of the signal e(t), e.g., its velocity, acceleration,jerk, jounce, absement, absity, abseleration, abserk, absounce, and/orthe like.

The time-derivative of displacement is the rate of change ofdisplacement, and is called velocity. In magnitude or absolute value, wehave instead the time-derivative of the distance is speed.

The time-integral of displacement, i.e. the area under thedisplacement-time graph, is known as absement.

Thus, to a PID controller on the “depth” sensor may, in oneimplementation, include computing depth, (e.g., in meters, centimeters,and/or the like), as well as its derivative in (e.g., meters-per-second(m/s)) and its integral in (e.g., meter-seconds (ms)), the velocity andabsement, respectively. In one implementation, integral and/orderivative may be computed on the depth error, e(t).

Processor K_{−1} is the absement processor, which operates on theabsement signal (the time integral of e(t)). Processor KO operates one(t). Processor K1 operates on derivative de(t)/dt.

In one implementation, the controller uses absity (integral of theintegral of displacement, i.e. the integral of absement) andabseleration signals, as well as acceleration signals. In someimplementations, higher derivatives and integrals such as jerk, jounce,abserk, absounce, and/or the like may also be employed.

Processed kinematics signals 2C40 are added in adder 2C45, and this sumis the MV (Manipulated Variable) or SV (Sculpted Variable) denoted,mathematically, as the signal u(t), denoted as signal 2C50 in FIG. 2C.

Kinematic signals are depicted with range or position as the basevariable, but this is in no way meant to be limiting. Kinematic signalsmay alternatively manifest as quantities 2C55 (e.g., momentum and itstime-derivatives) and/or as quantities 2C60 (e.g., action and itstime-derivatives).

Thus, the gesture-sensing or toposculpting system in the Spaceglass mayuse principles of physics as a basis for physical modeling andunderstanding of physical reality in the context of augmediated realitymanipulation or sculpting.

FIG. 2D depicts a toposculputing system in one embodiment.

FIG. 2D depicts an embodiment in which a drawing tool 201 is used todraw a two-dimensional manifold in a three-dimensional space, e.g., byextrusion. The drawing tool 201 may, in one implementation, may includean array of light sources, each individually addressable. In oneimplementation, each light source is addressable as to color, such asusing PWM (Pulse Width Modulation) on four color channels: R (Red), G(Green), B (Blue), and I (Infrared). In one implementation, each RGBIsource receives a PWM input from a buffer, such as an eight-bit bufferfor each color channel, e.g., so that each color is determined by acolor word, the color word being 32 bits long. This providesapproximately 4,295 million colors (or 4,096 Megacolors) that each lightsource can express, of which 16 Megacolors are unique in the visiblespectrum.

The drawing tool 201 may, in various implementations, take the form of asword or saber or virtual weapon-like object for playing games like avirtual battle. Alternatively, drawing tool 201 may be a flexible itemlike light rope, or it may be wearable, such as taking the form of anLED glove, a glove with retroreflective beads responding back passivelyto the active (extramissive) vision system of one or more extramissivespatial imaging digital eye glass devices, and/or the like.

The drawing tool 201 depicted in the example of FIG. 2D has 48 lightsources along its length, each of which contains, in one example, fourchannels, thus there being a total of 192 light sources therein. As thedrawtool is moved through space, e.g., in a long-exposure photograph,leaving behind exposure 2D40, which is shown to the left of the drawingtool 201. The topmost eight of the 48 lamps of the drawing tool 201 areshown magnified, to the left of the exposure 2D40.

This magnified depiction of these eight lamps is shown, labeled as thelamp L1, which makes the exposure E1, and the lamp L2, which makes theexposure E2, as both the lamp L1 and the lamp L2 are turned on at thispoint in time. The lamp L3 is off. The lamp L5 is on, and the lamp L8 isalso on, both also leaving behind their respective trace or streak orsmear of light, like the exposure El and the exposure E2 from the lampL1 and the lamp L2, respectively.

The lamps 2D8, of which the lamp L1 through the map L8 are examples,leave behind streaks of light, as the exposure 2D40, which, when thelamps 2D8 are controllably modulated, creates a controllable andaddressable two-dimensional manifold exposure. This exposure can, forexample, spell out words, text, graphics, or shapes (e.g., the nose coneof a rocket engine), useful for 3D visualization, for example. In theexample of text, we can spell out words (e.g., “HELLO WORLD”) in space,during a long exposure photograph, or in video, displayed asexposure-integrated video (e.g., photoquantigraphically).

As the drawing tool 201 is moved from left-to-right across the page, itleaves behind exposure 2D40, which gets longer and longer as the drawingtool 201 is moved further and further across the page.

At some point in the “HELLO WORLD” message, a first character, theletter “H”, will be “painted” or “drawn” through space, as for examplethe character exposure 2D51 from a font character generator set, whichmay also include “fuzzy fonts” (anti-aliased fonts or the tru-type™fonts) or full color pictures created in 3D space, as desired.

The set of characters that spell a message or alphabet represent acharacter exposure 2D60 of which the letter “H” (a character exposure2D51) is an example. The drawing continues through space, spelling outeach letter, with darkness (zero exposure) between the letters, or insome implementations, a background exposure between the letters so thatthey appear to hover on a 3D ribbon winding through space, first the“H”, then the “E”, and so on, eventually reaching the fifth letter, “O”as a character exposure 2D55, and then continuing with a space, followedby the letters “W”, “O”, “R”, “L”, “D”, and so on. In someimplementations, drawings, text, and/or the like may trace a ribbonaround a house, down the stairs, out of the front door, down the street,and/or the like.

Characters may, in one implementation, be drawn in a Video Orbitsstabilized space, as, for example, the person holding the drawing tool201 is wearing a DEG (digital eye glass) such as the apparatus 1300 thatcan see the drawing tool 201 as well as a surface such as the tabletopsurface 2D70.

The drawing tool 201 emits visible light for a character exposure 2D60,and this visible light does not interfere with the tracking by way ofthe DEG. In one implementation, the rays 2D21 of infrared light from theapparatus 1300 illuminate drawing tool 201 and then return as the rays2D22, to be sensed, irrespective of the visible color (RGB) emitted bythe drawing tool 201.

Visible rays 2D23 also illuminate the drawing tool 201, and arereflected as rays 2D24 to be recorded by the apparatus 1300. In oneimplementation, the apparatus 1300 captures video of the drawtool movingthrough space, while recording the exact position of the drawing tool201 relative to apparatus 1300 as a function of time. The apparatus 1300also records the position of the tabletop surface 2D70 relative to theapparatus 1300 as a function of time. Knowing the location of thetabletop (or any other recognizable object in a static scene), as wellas the position of the drawing tool 201, each with respect to theapparatus 1300, the relative position of the drawtool with respect tothe environment may be determined.

When using the drawtool throughout a space, an environment map may, inone implementation, be constructed of the space, or be downloaded (ifavailable) so that the drawtool can be moved around in the space,whether it be an outdoor landscape, or an interior space in a building,where drawtool can be taken room to room in an empty house and used todraw furniture in mid-air, e.g., so that a user can visualize what thehouse would look like furnished.

In the latter example, the user 180U does the Sightpainting™ movement(which is a combination of lightpainting and “painting with looks”) tosculpt and create their own reality. In one implementation, effectorsignals, projections, and/or the like may further be employed to mark,annotate, illustrate, and/or the like the viewed environment. Effectorsignals may be custom-made, made in real-time, drawn and/or derived fromstored records, and/or the like.

In one implementation, a second user 181U can see the abakographs ortoposculptures, such as those drawn by the user 180U, can annotate them,add to them, and/or the like.

This may be employed for various forms of collaboration such as 3Dvisualization, designing a rocket engine together, assisting a patientin a hospital, and/or the like.

In one implementation, another form of collaboration employingSightpainting may be gaming using, for example, the Unity 3D™ gamingenvironment.

For example, user 180U writes the words, e.g., “HELLO WORLD”, throughspace and assigns them a red color. User 180U has thus self-identifiedas red.

The long-exposure photograph of the words HELLO WORLD appears to hoverin 3D space as user 180U can walk around and view the abakograph of thecharacter exposure 2D60 from different angles.

The user 180U can also see this abakograph.

In one game scenario, the user 180U is a “red player” and the user 181Uis a “blue player”.

The red player tries to “paint the world red”, and the blue player triesto paint the world “blue”. The red player can touch and even passthrough “red paint” (red-colored abakographs) but if the blue playertouches an abakograph that is red, the blue player “dies” and loses thegame.

Likewise, the blue player, the user 181U, creates the exposure 2D90.Whereas the red player has chosen the “VOXaber of Death”™ system, asword like object with 48 LEDs (light emitting diodes), the blue playerhas chosen the “Blue Rope of Tyburn” (TRADEMARK) system which is an LEDlight rope that the user 181U has looped into a ring or “noose” toextrude (draw) “Blue Pipes”, such as the exposure 2D90, through space.

The players actually both use the same drawtool or type of the drawtool,including flexible light rope that can be attached to a “sword” made offoam rubber (soft so children don't get hurt while playing), or detachedto make other flexible shapes. In another implementation, players mayuse different draw tools and/or types of draw tools within a singleapplication instance.

In one implementation, the drawtool may be included as a promotionalitem with each DEG sold.

In one implementation, persons not wearing the DEG of the apparatus 1300can see the words like “HELLO WORLD” if the user 180U continually wavesthe drawtool back-and-forth. This may help entice non-participantobserves to join the game, either by buying or renting Spaceglasses. Forexample, bystanders will see players waving the lights through space andwant to play. A live video feed of some of the abakographs can also beprinted to help get people interested in playing, so they are morelikely to want to rent the Spaceglasses and join the game. Dedicatedplayers may purchase their own Spaceglasses and get the Xaber™ systemlike the drawing tool 201 they can keep. Thus, the embodiment of FIG. 2Dcan be used as a method of marketing the DEG of the apparatus 1300.

Additionally, the game promotes fitness, for example, as the red playermay have imprisoned the blue player in a room by writing “HELLO” acrossthe doorway. The blue player would die if he touched the red text“HELLO”, but may, for example, crawl under the right-hand side hand sideof the exposure near the character exposure 2D55, where it has liftedoff the ground by a small amount.

Crawling around and fitting through tight spaces helps develop bothstrength and flexibility, and games like Limbo Dance can be played withabakographic lines drawn through space that are visible in theaugmediated world.

In another game, various players may draw pipes through space. A bluepipe such as the exposure 2D90 belongs to the blue player. A user ischallenged to crawl or jump through the pipe, without touching thesides, and/or then to extrude the pipe longer, each time. The game maybegin, for example, with a “hoop” or ring, which gets “pulled out”longer and longer, until all but one player, taking turns, has eitherfallen or touched the virtual space occupied by the pipe.

In another game scenario, players throw a ball through the pipe without“touching” the sides of the pipe.

In one implementation, the Sightpainting™ system may be achieved via thefollowing: the apparatus 1300 senses reference points in theenvironment, while also tracking the position of a drawing tool 201. Letthe comparametric coordinate transformation from the drawing tool 201 tothe apparatus 1300 be g_{2,1}. Let the transformation from the apparatus1300 to some surface in the environment like the tabletop surface 2D70be g_{1,0}. The transformation from the drawtool to the environment isthus given by g_{2,0} g_{2,1} g_{1,0}.

Moreover, the position of every LED (light emitting diode) in thedrawtool, or a selected subset thereof, is recorded as a function oftime, along with photographic or videographic images, e.g.,high-resolution, of the light streaks of the exposure. Then image basedrendering is used to render the abakograph in the coordinates of theposition where the user 180U is standing, and specifically, the POE(Point-of-Eye) for each of the user's eyes.

In one implementation, this is performed and updated in real time (e.g.,continuously, periodically, on the basis of triggers, and/or the like)as users move through space, and also for other participants, such asthe user 181U, whose instances of the apparatus 1300 senses the ray 2D32and the ray 2D34 from the lamps 2D8 as well as their instances of theexposure 2D40.

The drawing tool 201 used to write “HELLO WORLD” may be marketed andsold, and may be referred to herein in various implementations via alight stick, a voxel stick, a pixel stick, a light sword, a VOX STIX™system, a PIX STIX™ system, a LIGHT SABER™ system, a Pixel Saber™system, a Pix Saber™ SYSTEM, a PlXaber™ system, a VOXaber™ system, aLIGHT SWORD™ system, a LightBroom™ system, and/or the like.

In one embodiment, a method of marketing DEG (digital eye glass) mayinclude: (A) offering the DEG for sale; (B) providing a free instance ofthe drawing tool 201, or the loan of a drawing tool 201 as part of apromotional offer, with the DEG; (C) inviting customers to participatein a competition using their drawtool, so-provided; and (D) providing anincentive as an outcome of the competition, thus, indirectly, promotingthe DEG product with the drawing tool 201.

In one implementation, when multiple participants are toposculpting ormaking abakographs, the visible-light recordings from various angles,e.g., from each of separate instances of the apparatus 1300, may becombined to capture more information, and render better abakographs.

Moreover, in some implementations, one or more auxiliary cameras,sensors, light sources, projectors, and/or the like in the environmentmay be used. For example, in a game space, a surveillance camera may beused from overhead, while the individual players provide their ownsousveillance recordings of the activity in which they are participants.The combination of surveillance with sousveillance provides a veillancerecording of greater accuracy than either alone.

The cameras may, in one implementation, operate in different modes. Forexample, the surveillance camera provides high resolution visible lightrecordings, whereas the sousveillance camera provides visual locationinformation in the “arm's length” close-range accuracy.

Frequency-Division Multiplexing

In some implementations, the drawing tool 201 can operate in a spectralband that is different from the “getting” in which the apparatus 1300senses position and/or range of objects.

For example, in one implementation, the drawtool can emit visible lightthat does not contain much energy, power, and/or the like in theinfrared portion of the spectrum used by the apparatus 1300 to determineits position. In one implementation, two users of the apparatus 1300 mayuse frequency-division multiplexing (e.g., via two different frequencybands) so as to not interfere with each other's sensing, nor beinterfered with by the drawtool unless the drawtool emits in theinfrared (e.g., so the apparatus 1300 can use passive infrared vision tolocate it).

To the extent that different players in a game, for example, might usedifferent colored visible lights, such different colors also embody thefrequency-division multiplexing that may be employed by gaming systemsdisambiguate the draw tools of various players.

Alternatively, time-division multiplexing may be used, as for exampleone drawtool can emit during a first “getting” and another drawtool canemit during a second getting, where the gettings are interleaved orinterlaced exposures from two or more instances of the apparatus 1300.

Moreover, the principle of lightspace may be applied (e.g., theprinciple of superposimetric imaging) to determine the resultantexposure due to each light source.

In this situation, the sources may be disambiguated in lightspace.Moreover, each light source within the drawing tool 201 is modulatedseparately so that the light due to that source is sensed, such thatindividual pieces of a toposculpture can be separately manipulated(e.g., individual “threads” of a pipe mesh or pipe sculpture).

FIG. 2E depicts further details of the toposculpting system in oneembodiment.

FIG. 2E illustrates various examples of abakographs, such as theabakograph 2E10 which begins with a weak exposure 2E11, as the usermoves a light source (or fingertip) through space in a spiral, whileincreasing the exposure toward the exposure 2E12, and eventually to theexposure 2E13 where the exposure is much stronger. In variousimplementations, variations in exposure may be user-controllable (e.g.,via a drawtool interface, trigger, virtual trigger gesture, pre-setexposure profile, and/or the like), automatic, preset, environmentallytriggered, and/or the like.

In one implementation, a trigger may be a 60 ohm rheostat, e.g.,salvaged from a toy race car, squeeze trigger that makes the car gofaster when squeezed harder. In this way the drawtool is fashioned as atoy gun with a light source that glows more brilliantly as the triggeris squeezed harder. The light is continuously adjustable in output.Thus, the draw tool as a “draw gun” can be drawn from a holster and thenused to draw shapes in space that look like beautiful lightpaintingsthat can be seen from any angle rendered in 3D space.

Alternatively, in an implementation recognizing the fingertip, theeffect may be synthesized as the thumb is moved toward the forefinger,e.g., to increase the intensity of a virtual light source. The finger180F itself is the drawtool, and rather than using a 60 ohm rheostat,the thumb 180TT is brought closer to the finger, with adjustment to theintensity of the virtual lightsource correlated with a detected distancebetween the fingertip and the thumb tip, an angle between the finger andthe thumb 180TT, and/or the like.

In some implementations, as the abakograph is drawn, a clenching gesturecan grab it. In one such implementation, the apparatus 1300 detects aflesh-on-flesh contact, e.g., as hands are clasped or fingers clasped.

Flesh-on-flesh contact may be sensed, for example, by 3D position inspace, by the deformation of the flesh that happens when it is pressedagainst other flesh, and/or the like, confirming and sensing the degreeof pressure, which forms a continuously variable gesture. Continuouslyvariable gestures are useful in this context as parameterizedcontrollers like, for example, the light dimmer gesture in which thelamp output is proportional to the angle formed between the finger 180Fand the thumb 180TT, which shall be called “angle FVT” taken by theapparatus 1300 as the angle of the finger 180F, through the visionsystem 180V, to the thumb 180TT.

In one implementation, the cosine of angle FVT may be computed and usedfor adjusting the duty cycle of a PWM driver to the light source, and/orotherwise used to control the level of the virtual light source.

In the case of the virtual light source, the effects of illumination aresynthesized (e.g., rendered), such as to capture room illumination bythe light source and its instance of the abakograph 2E10.

Various kinds of abakographs can be manipulated. For example, apipe-shaped instance of the exposure 2D90 is opened up or bent orre-sculpted to the abakograph 2D91 using hand gestures. This polygonextrusion can also be unwrapped into the abakograph 2D92.

Various forms of computer graphics and image-based rendering providephotorealistic sculptures that have high visual appeal, and can bemanipulated as art objects, game objects, or for practical and/orindustrial purposes, such as designing the pipework of a city, sculptingand building a virtual musical instrument like a hydraulophone fromvirtual pipes using virtual water flowing through the pipes, which canbe toposculpted in a CFD (Computational Fluid Dynamics) environment,and/or the like.

FIG. 2F depicts a hand-based toposculpting mesher in one embodiment.

FIG. 2F shows an embodiment using the finger itself as the drawtool. Thefinger is fitted, in the illustrated example, with an LED glove such asa “Glove Light” toy. In another implementation, a projection is made onthe finger with a light source, which may include detectable structuresprojected onto the finger, may reflect from reflectors affixed to thefinger or a worn glove, and/or the like. In one implementation, is thedrawtool may be effectuated synthetically (e.g., just by finger trackingalone). Let's consider the situation where lights are present (in in thebare hand version, we can consider them as virtual lights).

In one implementation, a linear array of eight LEDs is present (orsynthesized), and they lay down an abakographic mesh, e.g., defined byan eight by eight (8x8) array of 64 points in the lamps 2F64. In oneimplementation, the points are modeled as spats and can represent, forexample, surveillance cameras and their vantage points, thus allowing ahand gesture of the hand 180MD to sweep out and define a surveillancemesh, or mesh city, or other mesh space.

The mesh may include eight glowing exposures such as the exposure Elfrom the lamp L11. The eight instances of the lamps 2F8 define, at afirst point in time, the lamp L11, the lamp L21, the lamp L31, . . . ,and the lamp L81.

The lamp L11 is moved along to generate the exposure El, and it turns onand off or varies in output to create an abakograph. At some point intime later, it reappears as the lamp L12, then the lamp L13, and so on,appearing eight times to make eight different parts of the exposure E1.

The lamp L21 also reappears as the lamp L22, then the lamp L23, and soon. Each lamp, all the way down to the lamp L81, traces out a path. Forexample, e the lamp L81 traces out as the lamp L82, then the lamp L83,as the lamps are considered at various points in time.

In the illustrated implementation, across from left-to-right there istime, and from top to bottom, there is space.

The eight lamps thus trace out a space-time continuum that is sampled(discretized) in space, but almost continuous in time, though we'vechosen to sample it as an 8 by 8 (i.e. 64 pixel) array in theillustrated example. The means, apparatus, device, software, firmware,or the like that performs this task of carving up time to match space iscalled a Mesher™ method or system. The Mesher allows us to generate a 3D“wireframe” from an abakograph. When applied to a row of parallel lineslike the lined paper of a schoolbook, the Mesher generates what lookslike graph paper from the lined paper. In our space-time continuum, theMesher, in one embodiment, determines the time increment that matchesthe space increment, so that the axes of the mesh are isotropic. By wayof analogy, in one embodiment, abakography generates an object in 3Dspace that resembles a beadless abacus. The Mesher inserts virtual beadson each “wire” (exposure streak-line) of the abacus-like exposure, insuch a way that the spacing between the beads matches the spacingbetween the wires. Then the beads are connected together in a directionperpendicular to the wires, in order to therefore generate the wireframemesh.

In this way, the finger is able to make “fingerpaintings” in space withreal and/or synthesized light sources, to, in one embodiment, generatewireframe models or meshes, which can then be texture-mapped or“painted” with light or with hand gestures, or the like.

Thus, in one implementation, a mesh is defined directly in time, andindirectly in space (e.g., inferred, temporally).

As video is captured of the sweep across, the content can be edited byvarying the light output (such as synthetically) in post-production, toachieve any desired pattern after-the-fact.

In one implementation, the user of the digital eye glass 190 may use adrawing tool 201. Each user may track the instances of the drawing tool,and the two or more users may thus experience a shared drawing space(e.g., in the first augmediated-reality space 1000 of FIG. 1A) via thedigital eye glass 180 and the digital eye glass 190, without having tocreate any physical markings during user interaction and/or after userinteraction is completed. A recording can be made of the collaborationsession, so that the recording may be played back and thus allows theusers to review the collaboration session for further study.

This form of computing creates a natural user interface that is free ofthe need for metaphors, and thus, in one implementation, forms a directuser interface experience.

FIG. 2G depicts an embodiment of an inverse surface and a meta table,which in some implementations may be referred to as a METAtableSousface™ system.

FIG. 2G depicts an inverse surface and a meta table in one embodiment.In one implementation, whereas smart surfaces may be controlled bysurveillance cameras (e.g., cameras affixed to property, architecture,walls, ceilings, boards, and the like), an inverse instance of thesurface 130 may be configured and/or optimized to be tracked by awearable camera system, such as may be worn, for example, by the user180U.

In one implementation, gesture tracking may be performed against anobject, such as one under sousveillance. In one implementation,sousveillance is the recording, sensing, capturing, and/or the like ofan activity by a participant in the activity. In one implementation, thewearable camera is a sousveillance camera, not a surveillance camera,and thus, may provide human-centered coordinates, such as forhuman-centered interaction design.

In one implementation, gestures performed on or against an objectprovide a meta-tactile experience and/or information, e.g., where metainformation may be affixed, marked, displayed, designated, associated,provided, and/or the like on and/or to the object and/or relatedobjects. In one implementation, such experience and/or informationprovides haptics (e.g., touching) and/or autohaptics (e.g., touching ofthe self, sensing when one's own hands come together, and/or the like).In some implementations, haptics and metahaptics combined with gesturesensing may be referred to as meta haptics, where meta informationexists at the nexus of haptics and gesture sensing. In someimplementations, haptics with one's own body, e.g., touching the handstogether, touching the index finger and thumb while touching somethingelse with the middle fingers, and/or the like, may be referred to as“autohaptics.” For example, in one implementation, an autohapticmulti-touch gesture may include clenching the thumb and index fingertogether while touching an object with the remaining three fingers,which form a triangle by the fingertip points, having thus, in oneimplementation, nine degrees of freedom in 3D space. In oneimplementation, a tenth degree of freedom may be defined by how hard thethumb and the index finger are squeezed together. In one implementation,a closed flesh loop made by the thumb and index finger may provide inputand/or additional parameters for a wide variety of autogestures. In someimplementations, an object amenable to meta haptics may be referred toas “metable” and one that is not as “immetable.” In one implementation,objects which are beyond a user's reach, touch, grasp, and/or the likemay be immetable, e.g., the stars and distant planets, the sun, themoon, the objects 2G140 high near the ceiling in a building with veryhigh ceilings, and/or the like. In one implementation, objects slightlybeyond reach like the objects 2G150 may be referred to as“semi-mettable,” and may be reached with a meta wand, such as thetoposculpting wand 2G160, the drawing tool 201, and/or the like.

In one implementation, immetable objects may be interacted with, e.g.,virtually touched with some haptic sensation, though less, in oneimplementation, than for metable objects. The toposculpting wand 2G160can, in one implementation, be fitted with a geophone, vibrator, pagermotor, the Tactaid™ system, and/or the like, for example, to conveyvirtual haptics from a distant object being “wanded.” For example,holding up the wand and looking through the ring 2G161, e.g., at adistant galaxy, the wand may vibrate when the Andromeda galaxy ispositioned in the ring 2G161. Thus, the ring 2G161 can, in oneimplementation, operate as a haptic viewfinder, allowing users, forexample, to “touch the stars” or otherwise “feel” the sky, near ordistant objects, images on a billboard, advertisement, document orcomputer screen, and/or the like. In one implementation, geophonic bodytaction may be implemented in the apparatus 1300 such that a user canview the stars and feel certain ones as vibrations in or on the head. Inanother implementation, the handle of the toposculpting wand 2G160and/or the apparatus 1300 may be fitted with electrodes configured toproduce direct and/or indirect stimuli (e.g., mild, non-painful electricshock) and/or other forms of stimuli, such as to create a sensation oftaction in association with remote or distant objects.

In one implementation, the toposculpting wand 2G160 has a user graspableportion that is or has some controls on it. For example, e the usergraspable portion can be and/include a trigger, which, in oneimplementation, may include a variable resistor. Such trigger rheostats,which come in resistance values of 15, 25, 45, 60, and 90 ohms, may bewired in series with an illumination source, and/or may be sensed by theapparatus 1300 and/or one or more microcontrollers associated withtoposculpting wand 2G160 for control of its functions, such as its lightL1, light L2, light L3, etc.

In one implementation, the toposculpting wand 2G160 is a toposculptingwand. In one implementation, the toposculpting wand 2G160 may be usedfor gaming as a “magic wand.” In one implementation, the toposculptingwand 2G160 may be used as a sculpting tool, e.g., to shape variousobjects in 3D space. In one implementation, the toposculpting wand 2G160may be used for lightvectoring (e.g., painting and sculptinglightvectors), e.g., as a visual art form, as a way of visualizingobjects, designing objects, 3D visualization, navigation, and/or thelike.

In one implementation, the toposculpting wand 2G160 is (includes) a ring2G161 or other shape and/or arrangement of light sources, such as coloraddressable LEDs (Light Emitting Diodes). The ring 2G161 includes thelamp L1, the lamp L2, the lamp L3, etc.

In one implementation, the toposculpting wand 2G160 has a handle 2G164,such as may include a detacheable grip 2G163 which can be used to spinthe shaft 2G165 of the toposculpting wand 2G160. In one implementation,this ability to spin the wand generates spheres in long exposureabakographs such as the abakograph 2G168. Spherical abakographs can thenbe “grasped” in 3D space and can be manipulated, e.g., as “primitives”to design and build other objects.

In one implementation, spherical abakographs like the abakograph 2G168can also be generated using a metable circular object 2G130, such as adinner plate, saucer, disk, and/or the like, where a grabbing andspinning gesture is recognized through the sensing and display apparatus1300, point clouds are determined and extruded, and/or the like. Themetable circular object 2G130 may be called a metable object. In oneimplementation, sensing and display apparatus may employ specific,specialized and/or dedicated effector and/or sensor signals to generatespherical bakographs from the metable circular object 2G130.

Other metable objects like the object 2G120 may, in someimplementations, be used to generate abakographs. In one implementation,metatouch is recognized, e.g. if the object is touching the surface 130,if the object is touching the hand 180MS, the hand 180MD, and/or thelike. In one implementation, different gestures may be assigned when anobject 2G120 is touched by one hand than when it is touched by one handthat is touching another hand (e.g. both hands touch it while the handsare touching each other, or one hand touches it while the other handtouches the first hand without touching it, and/or the like). Theseexamples of meta haptics form unique gestures that have unique effects.

For example, a user holding the toposculpting wand 2G160 in one hand180MS touches a dinner plate or other object with the other hand 180MD;this gesture may, in one implementation, signify that the user issignaling abakography, and thus the dinner plate is recognized as anabakographic object for meta-bricolage. In one implementation, metabricolage is the making of virtual objects from real objects, e.g.,sitting on a table, collected in a work environment, collected whilewalking around, and/or the like.

In one implementation, a rocket engine, an exhaust manifold for amotorcycle, a pipe sculpture, and/or the like may be designed bygrabbing a dinner plate or the metable circular object 2G130, spinningit to make a sphere, moving the dinner plate through space to make apipe (extruding a cylinder), grabbing the abakograph 2G168 (such as asphere object, whether generated by the wand, the dinner plate, by handgestures, and/or the like, or recalled from a library of 3D objectsusing hand gestures) in 3D space, putting the two objects together,and/or the like.

In one implementation, some shapes, such as spheres and cylinders, maybe pulled from a library of shapes. For example, a hand gesture such astouching a dinner plate with the left hand 180MS while making a drawingshape with the right hand 180MD, may be recognized as a “draw circle”(e.g. draw something like this dinner plate) gesture. In oneimplementation, the apparatus 1300 and or components may retrieve animage of the touched object, compare it with records in a shape library,and select one or more closest matching shapes. In one implementation, auser may be presented with a plurality of selectable shapes in relationto a touched object, such as where the comparison between the objectimage and stored shape records yields results differentiable by lessthan a threshold.

In some implementations, more complicated, intricate, and/or compoundshapes may be used and/or created. For example, the user 180U can pickup the dinner plate and wave it through space to make, e.g., a manifold2G184. The manifold 2G184 be may a curved pipe-like shape. For such amanifold, the user 180U may have waved the dinner plate in a circulararc, e.g., to generate a “sweep” pipe like the gently curved greyplastic elbow fittings used for electrical conduit.

The manifold 2G184 may, in one implementation, be generated by waving adinner plate, roll of duct tape, Frisbee, and/or the like round objectthrough space in an arc. In one implementation, such manifolds may becreated with the toposculpting wand 2G160.

In some implementations, manifolds may be shaped and stretched and movedby reaching for, grasping, grabbing, poking, and/or the like in 2Dand/or 3D space. In one implementation, when hands touch both ends of apipe or sweep or other similar object that has two ends, the object ismade to glow a different color, to flash, to flicker, and/or the like,such as to indicate that it has been selected, e.g., the selected objectbeing a manifold 2G189. Once the object selected is highlighted in thismanner, hand gestures are authorized for, recognized for, and/orassociated with curving or bending of the pipe, such as using thecurvature of the fingers to indicate desired curvature of the pipe end.In one implementation, curved fingers of the hand 180MS indicate that atighter curve radius is desired by the user 180U, so the end of the pipeto the user's left (to the viewer's right in FIG. 2G) is rendered with atighter curve, than the other end of the pipe which is “straightenedout” in response to an “unwrapping” hand gesture.

In one implementation, videographic abakographs may be edited using aCEMENT (Computer Enhanced Multiple Exposure Numerical Technique) thatincludes operations, such as: operation (1) including capturing objectdata, as video, images, location data, and/or the like of an objectswept through space; and operation (2) including segmenting the objectdata to make an abakograph. In one implementation, when using thetoposculpting wand 2G160, the segmenting is automatic, e.g. if the lightL1, light L2, light L3, etc., are modulated and sensed by a lock-incamera, vision system, and/or the like capable of ignoring all but theselights, e.g., using the principle of Superposimetric analysis andLightspace). Operation (3) includes displaying the edge or rim extrusionof the object integrated over time of a particular gesture getting. Inone implementation, the getting is started and stopped with handgestures, or in the case of the wand, the getting may be started andstopped by squeezing and releasing a trigger 2G162. In oneimplementation, the getting has a soft start and a soft ending, e.g. byeasing the trigger on slightly while moving the toposculpting wand 2G160so as to “feather” the start or end, as shown in the manifold 2G181which starts softly on the left, and ends abruptly on the right, whereit “joins” the manifold 2G184. In an implementation not using the wand,an un-digital multifinger gesture can be used, such as varying the anglebetween the thumb and index finger to create a continuous trigger that“feathers” the beginning or ending of an abakograph. In oneimplementation, abakographs formed by light L1, light L2, light L3,etc., are integrated into toposculptures by interpolating the points andcross-meshing to create a “wireframe” model that is filled in like acontinuous curved pipe or the like.

In one implementation, joins between the manifold 2G182, the manifold2G181, and the manifold 2G184 are made by “pushing into” the manifold2G184, e.g., drawing the manifold 2G181 or the manifold 2G182 toward themanifold 2G184 with one hand 180MD while making a first with the otherhand 180MS to indicate that the manifold 2G184 should have a hole“punched through” it to receive a pipe connection to another pipe.

In one implementation, once the pipes are connected, a fluid flowsimulation may be run, e.g., to check for leaks, connections, and thelike. For example, when pipes are joined, test fluid is rendered in themof a particular color to show which pipes are connected together in aplumbing circuit.

In one implementation, flowing gently curved pipe circuits may be builtand/or generated as whimsical sculptures, practical plumbing, electricalconduit installations, such as may be designed, simulated, and/or testedusing the Bricologic™ system. In one implementation, bricologic islogical (computational) bricolage, e.g., tinkering, using a variety ofobjects, including everyday household objects, industrial and/or designobjects, architectural elements, specially designed and/or taggedobjects, and/or the like to build things, e.g., by using the objects formeta gestures, metasculpting, abakography, toposculpting, and/or thelike.

For operation (4), the following applies: in one implementation, a pipelike the manifold 2G189 (curved pipe) is rendered as aphotoquantigraphic summation of exposures, e.g., over a certain segmentof exposures. Such a manifold 2G189 (such as a pipe) may be created, forexample, using the toposculpting wand 2G160, e.g., swept from the user'sleft hand 180MS to the user's right hand 180MD, by a dinner plate,and/or the like object moved accordingly. In an implementation, wherethe user 180U moves hand 180MS toward the center of the pipe, e.g.,wishing to shorten it (“squeeze it inwards”), video frames are removedfrom the beginning of the recording, so the sum of integrated exposureframes starts later in the sequence. By way of example, an abakographmade from a ten second video exposure will have 600 frames (60 framesper second times 10 seconds). The full exposure is a photoquantigraphicsum from frame 1 to frame 600. Shortening the pipe, at the end of thepipe corresponding to the user's left hand, involves operation (4 a)using gesture recognition and/or hand-position recognition, to determinewhere the hand has moved inward along the pipe; operation (4 b)including calculating space-time compensation to convert this change inspatial position to a change in time (e.g. to compensate foracceleration in the original exposure), thus converting the spaceposition to a time position, such as, for example, left hand movesinwards one third of the distance along the pipe, which, for example,corresponds to one quarter the way through the video; operation (4 c)including converting the time position to a frame number in the video,which here would be, for example, frame number 150; and operation (4 d)including computing a new photoquantigraphic sum from video frames 151to 600, and displaying or rendering this sum as the new abakograph ofthe pipe, which no longer includes a contribution from frames 1 to 150.

Likewise, the right hand 180MD may control the ending frame of thesummation of integrated exposures. For example, moving the right handtoward the center might change the sum so that instead of going fromframe 150 to frame 600, now it might go from frame 150 to frame 400.Thus, a long exposure photograph may be generated as a streak of tracesthat starts and ends at the hands, and as the hands move, the streakand/or photograph is rendered so it matches the end points of the hands,e.g., by selecting the appropriate frame in the video sequence. Since,in one implementation, the vision system is true 3D, the result can berendered from any angle where the user is positioned. Also, one or moreadditional cameras may be affixed in the environment, e.g., to capturehigh resolution exposures of the wand or object moved through space andthus render it photo realistically if desired (e.g., so it actuallylooks like the long exposure photographs resulting from car headlightsmoving down a dark road for example).

For operation (5), the following applies: in one implementation, ameta-space-time continuum operator determines the relationship betweenspace and time, e.g., between space and frame number of the videorecordings of the abakograph. The video recordings may, for example,capture photorealistic traces and/or true 3D positions of all points onthe extruder object such as the surface 130 (also called an object) orthe toposculpting wand 2G160. When the extruder object is thetoposculpting wand 2G160, additional meta information may be recorded,such as the position of the trigger 2G162. In some implementations, theextruder may also be a printed letter, a gestured letter, a 2D image, anicon, a nozzle, an extrusion die, a “cookie cutter,” and/or the like.For example, if user 180U points to the letter “I” in a newspaperheadline, with the left hand 180MS, while pulling an extrusion gesturewith the right hand 180MD, an “I” beam is made, by extruding the letter“I”. Similarly, virtual “angle iron” beams can be made by pointing tothe letter “L” with one hand and extruding it with the other. In oneimplementation, computer vision can use the exact font of the letter,e.g., so that a user can use various real physical “I” shaped objectslike news printed fonts, or cutouts, children's letter blocks, and/orthe like, to rapidly make beam designs and test them, e.g., usingcomputer simulation of beam strength, stiffness.

In one implementation, a space-time operator determines which frame ofthe video corresponds to which position along the beam, so that, forexample, equal distances along the beam or other extrusion can belocated in the videos.

For operation (6), the following applies: in one implementation,cross-meshing is then applied, e.g., at equal points in space and/ortime. The traces shown in FIG. 2G are along the path of extrusion. Insome implementations, cross-meshing may be displayed as circles alongthe pipe, (in one implementation, as “beads” along each of the “wires”of a pipe-shaped abacus that make up the pipe, where each set of beadscorresponding to a particular point-in-time are joined with a “wire” toform a circle), e.g., one circle rendered at each unit of distance alongthe pipe, and/or spaced in accordance with a variable spacing profile.

In this way, a crisscross mesh is generated which may, for example, berendered as photorealistic glowing light having a pleasing look similarto long-exposure “lightvector painting” photographs generated by summingphotoquantigraphically the video frames.

In one implementation, metable objects such as dinner plates, boxes,cardboard which can be cutout with scissors, and/or the like may be usedby a user 180U to build, for example, a rocket engine, using glue, tape,scissors, and other objects. In one implementation, the vision system ofapparatus 1300 recognizes the glue, tape, scissors, and/or othercomponents, e.g., via image comparisons, to infer that the user istrying to build something, to engage instructional data and/orapplications, to determine what the user is trying to build, and/or thelike.

In one implementation, the metable sousface may be a surface 130 with anadditional apparatus 2G169 that assists in sensing toposculptinggestures by, for example, sensing when the surface 130 is touched, andsensing hand gestures thereupon.

In some implementations, the apparatus 2G169 senses touch upon thetable. This may be applied, for example, where the apparatus 1300 hasdifficulty sensing the difference between fingers that are close butaway from the table and fingers that are actually pressing the table. Inone implementation, the apparatus 2G169 is another instance of apparatus1300, e.g., worn by another user 181U, where the ability to “lookthrough the eyes of another person” grants a side-view. In oneimplementation, a camera on the table looking across may be employed.

In one implementation, the surface 130 includes a lattice of markings onit that are human and/or machine readable, e.g., so that the surface hasan array of dots or other shapes on it that help the user orient thehands on the surface. In one implementation, tactile dots may beemployed that, when touched, provide stimulus to the fingers, e.g.,direct (tactile) and indirect (e.g. by mild electrical stimulus which ishard to localize). Indirect stimulus allows, for example, an object toseem to be felt. Thus, in one implementation, a user 180U placing a lefthand on the surface 130 while touching a mid-air virtual object like themanifold 2G189, feel a “tingle” while touching it, such as via mildelectric shocks, well below the pain threshold, and/or that have aquality of poor localization (e.g. one can't tell where exactly it iscoming from). Alternatively, a clicking sound in the headgear of theapparatus 1300 and/or a vibration can be a cue that users will mistakefor touch. In this way, the manifold 2G189 can be made to seem very much“alive” when touched, e.g., seem to really be there in the world,because something of poor localizability happens when it is touched.

In some implementations, a table having a surface 130 (also called ametable object or a metable surface), for being sensed throughsousveillance may be referred to as a METAble Sousface™ system. In oneimplementation, whereas the surface of a shape, such as an auto body,may be its top, the sousface may be its inside, e.g., what would be seenif the body was taken apart and/or turned upside down.

In one implementation, a sousface is amenable to sousveillance, e.g., asurface that lends itself to be watched rather than a surface that doesthe watching (such as one with sensors). A sousface may thus, in someimplementations, be said to have and/or act as though it has inversesensors.

In one implementation, an epitrochoid (of which the epicycloid is oneexample) may be formed by rolling a small disk on the surface of alarger disk. In one implementation, a hypotrochoid (of which thehypocycloid is one example) may be formed by rolling a small disk on thesousface of a larger disk.

In one implementation, taking the limit of large radius of curvature,the hypotrocloid and the eptrocloid may become trocloids (of which thecycloid is an example). In such an implementation, a surface and asouface may merge.

In the described disk examples above, souface and/or surface may beconsidered one-dimensional. In a two-dimensional implementation,hypertrocloids may be constructed such as by rolling cylinders, cones,spheres, boxes, and/or the like on other cylinders, cones, spheres,boxes, trays, and/or the like, which may be referred to in someinstances as a “hyperroulette.” In one implementation, a hypercycloidmay be an example of a hypertrocloid that is traced out by a line alongthe outer surface of an object rolling on a surface and/or sousface(e.g., flat or curved). For example, a light rope, or retro-reflectivetape, or colored tape, or a colored line, running lengthwise along theouter edge of a cylinder (e.g., a cardboard mailing tube), rollingacross a table during a table during a long-exposure photograph may forman image, photographic document, and/or the like, which may be referredto in some instances as a Topoloid™ system or a Topoid™ system.“Topoloid” refers to an abakograph formed as per the above, or a meshedversion of the abakograph (i.e. its “wireframe” mesh). In oneembodiment, the meshed version is constructed by running the Mesher onthe abakograph. In this embodiment, a row of lights together with a lineconnecting them may be recognized by the apparatus 1300 so that thelights trace out the “wires” of the abacus, and the entire strip of tapeis used to draw each crisscross line at a particular point in time, thatmakes the crisscross lines have the same spacing as the abacus wires.This generates a mesh that resembles a chicken wire, so that thetoposculpture is a wireframe model that can then be texture-mapped asdesired.

In one implementation, topoloids may be employed within a bricolageapproach, including making virtual objects from hyperroulettes, e.g.,rolling things on other things while watching them through Spaceglassesthat record these actions.

In one implementation, a sousface may have a lip and/or edge, such as arim positioned around its perimeter (e.g., as in a tray, basin, and/orthe like). Such a sousface may be employed for constructing aBricoloids™ system and/or a Brichoid™ system (e.g., from long-exposurebricologic).

In various implementations, different objects may be constructed thusly,e.g., with hands, found objects, and/or the like rolled on a sousface,and/or with one or more different kinds of wands, to wand objects and/orthe sousface.

In one implementation, a toposculpting wand 2G160 may include a ring2G161 including a wheel, disk, and/or the like on a bearing that spinsor rotates, such as by way of a motor, rolling it along a sousface,and/or the like. In one implementation, a shaft encoder may be used toencode the space-time continuum of a long exposure video photograph. Inone implementation, a motor, actuator, and/or the like may be employedas a shaft encoder, e.g., to turn or sense the turning of the ring 2G161as it rolls along the surface and/or sousface. In one implementation,the lamp L1, the lamp L2, the lamp L3, etc., may each delineate and/orcreate a cycloid shape in a long-exposure photograph, such as made whilerolling the toposculpting wand 2G160 along a surface and/or sousface. Tothe extent that surface 130 may be basin-shaped, it may have availablevarious curvatures defined by a major axis (loose curvature), a minoraxis (tight curvature), and various other curves and undulations in thesurface. Thus surface 130 and the toposculpting wand 2G160 can be used,in various implementations, to generate a wide range of different kindsof hypocycloids as abakographs, e.g., when filmed, photographed,captured by video together, e.g., with 3D cameras of the apparatus 1300and/or other auxiliary cameras around the surface 130, and/or the like.

Other lights on the toposculpting wand 2G160, e.g. not right at theedge, but further in, may, in one implementation, define hypotrochoidswhen photographed in a long exposure. A flashing light gives a path ofthe hypotrochoid that looks, in one implementation, like a dashed curve,e.g., a dotted line bent in the shape of the hypotrochoid. In animplementation, where other shapes are attached to the toposculptingwand 2G160 for rolling along a surface 130, other topoloids/toploids maybe created and/or subsequently toposculpted, e.g., further using handgestures and other forms of interaction, to generate bricoloids, such asmodels of cities, pipework, pipe networks, space stations, rocketengines, and the like. In one implementation, a cone attached to thering 2G161 with a row of lights going from apex to base will generate afamily of curves in 3D space, including a curve parallel to the surface130 from the apex of the cone, a hypocycloid from the light at the baseof the cone, and various hypotrochoids from the lights there between.

In one implementation, the toposculpting wand 2G160 may also featureplanetary gears and/or other gears, such as that turn other parts of thewand at different speeds and at different angles of rotation. Forexample, the ring 2G161 may drive another ring at right angles to it, sothat, in one implementation, a helix is generated rather than ahypotrochoid. The two rings may also be adjustable, in oneimplementation, so their angle can be changed, e.g., to generate theabakograph of anything in between a helix and a hypotrochoid.

In one implementation, abakographs and/or other apparatus-generatedshape may be interpreted as electrical signals, allowing users to createarbitrary signals using the toposculpting wand 2G160 through, e.g.,long-exposure photographs and/or video graphs, sculpting the signals byhand gestures, and/or the like.

Thus, in some implementations, the apparatus 1300 may be used to sculptvarious objects, such as using hand gestures, hand gestures againstobjects, hand gestures with objects, the Bricology™ system of objects ina toposculpting environment, and/or the like. In one implementation, atable, the METAble and/or other similar table, surface, sousface, and/orthe like may be employed for one or more of these operations and/oruses.

In one implementation, a surface 130 (a table bearing surface) may bedenoted with a metable label 2G110 (such as a “METAble Sousface”), suchas in the example of FIG. 2G. In one implementation, the metable label2G110 is machine readable, e.g., on the sides of the table, the top halfof the label being white, in retroflective material, and the bottom halfbeing black, with retroflective letters. Labels, like the metable label2G110, may be placed around the edges of surface 130, e.g., so that itcan be recognized and positioned in 3D space even in total darkness.

In one implementation, total darkness may be afforded by squeezing thetrigger 2G162. For example, pressing the trigger can automatically turnoff the room lights, such as to make toposculpting easier, morebeautiful, and/or visually engaging.

In some implementations, even in the dark, the surface 130 may bevisible to the user 180U by way of the apparatus 1300 which may beconfigured to see in the dark, but additionally, the table may beconfigured with instances of the metable circular object 2G130 that areilluminated, e.g., to emit visible light during a blanking interval ofthe apparatus 1300. The metable circular object 2G130 may be calledsmart dots. The apparatus 1300 senses, in one implementation, at a framerate of 120 FPS (Frames Per Second) and the metable circular object2G130 may, accordingly, emit light briefly (e.g., for about onemicrosecond) during a time when the apparatus 1300 is not responsive tolight. Such synchronization allows for system control of the dots, sothey can be invisible or visible to the apparatus 1300, as desired. Inthis way, visibility to the human eye(s) and to the apparatus 1300 maybe separately controllable, optimized, and/or the like In someimplementations, the metable circular object 2G130 emit infrared light,e.g., to help localization tasks, deliberate exposure to the apparatus1300 in a visible getting, and/or the like. In one implementation, thereare thus three gettings and/or controls of the gettings; (1) human eyeexposure; (2) visible light camera exposure of the apparatus 1300; (3)infrared vision system camera of apparatus 1300. In one implementation,there is fourthly a retroreflective quality of the metable circularobject 2G130 that interacts with the apparatus 1300, allowing it tolocalize itself at the table. In yet another implementation, each of themetable circular object 2G130 is a proximity sensor to localize objectson the table, measure how far away they are, measure contact such as bythe hand 180MS and the hand 180MD, and/or the like. In anotherimplementation, the table is made of material, such as metal, sensitiveto contact with the user's hands, gloves, and/or the like. In anotherimplementation, the smart dots may include holes, e.g., that arethreaded for prototyping, so the user can screw objects down to thetable for tinkering. The holes can, for example, draw vacuum or provideair pressure like an air hockey table for floating objects to be slidaround or sucked down to the table and held there. In oneimplementation, the table is ferrous, permitting use of magnets as usedon optical tables or the like.

Thus, in one embodiment, the table may be used for bricologic and/orcomputational bricolage.

Geophones may sense touch and effect touch, e.g. the table vibrates toprovide vibrotactile haptics.

In some embodiments, the surface 130 may be designed and/or configuredspecifically for meta haptics. In one such embodiment, a Meta Table™system may be constructed having a pattern on it that is retroreflectiveto the vision system worn by user 180U. Such a vision system, as shownin one example in FIG. 2G, may contain two color cameras, a range camerasuch as the LiDAR unit 187, various other sensors, and/or the like.

The various other sensors may, for example, include a gesture band 2G190on the wrist of the hand 180MD. In one implementation, the gesture band2G190 senses self-gestures by movement, electrical conductivity of theself-skin, by impedance tomography, and/or the like. Thus, when themiddle finger 2G199 touches the object 2G193, the gesture may beinterpreted differently depending on whether index finger and thumb ofthe hand 180MD are touching or not. Gesture meaning may, in oneimplementation, also vary continuously as a function of how hard indexfinger and thumb of the hand 180MD squeeze together, which may bemeasured by a gesture band 2G190, e.g., using EMG (muscle signals),vibration, electrically, measuring band strain (e.g., caused by wristgirth) via one or more strain gauges, and/or the like. In oneimplementation, the vision system in the apparatus 1300 may beconfigured to perform this task, e.g., such that highly expressivegestures may be used, such as may be based on looping or closure ormating of flesh-on-flesh as another input variable to touch.

In one implementation, input variables such as these allow a user tofeel what's happening, objects being created and/or manipulated, and/orthe like. Meta taction, as afforded in the above examples, helps provideuseful tactile feedback through gestures that are mettable, i.e. onmettable subject matter, as well as gestures that are multiply haptic,like touching with one finger while squeezing other fingers or thumbtogether in various ways that can be felt by the user and understood bythe apparatus 1300 e.g., via processing apparatus 908.

In some implementations, a Meta Table may contain various sensors and/oreffectors selected and/or optimized to sense and effect haptics and metahaptics.

In some implementations, a bricological environment can be createdanywhere, not just using a table. For example, bricology may bepracticed outdoors on the ground, where the sousface is the ground. Theground can be fitted, for example, with geophones and/or other sensorsto aid in sousveillance with Spaceglasses or the like. In this contextsmart ground stakes can be driven into the ground. Such stakes usedand/or interfacing with Spaceglasses and/or the apparatus 1300 may, insome implementations, be referred to as sousveyor stakes, whichfacilitate a bottom-up (“sousvey”), rather than top-down (“survey”)hierarchy. For example, any wearer of Spaceglasses can participate inannotating reality, with the curation of properties and physical spacesno longer in the purview of only the surveyors and other officials. Inone implementation, souveyor data may include annotations, distances,angles, heights, horizontal locations, landmarks, and/or the like, suchas may be stored in association with one or more user identifiers and/orprofiles.

In one implementation, the resulting sousveillance creates aparticipatory world in which individuals can, for example, tinker,create virtual cities, augmediated cities where buildings can besculpted, and/or the like. In one implementation, even distressed ordiscarded objects, such as an old cardboard box from a dumpster or curbside, may be held up to position in view of the apparatus 1300, and mayhave its image manipulated via hand gestures to shape the box into abuilding, by copying shapes and stretching solid rectangular objects,and then “place” the building into an empty parking lot on the street toenvision a skyscraper there.

Thus, in some implementations, toposculpting through bricolology allowsdesign and manipulation of, for example, a city of the future, a spacestation, a “pipe dreams” musical instrument that runs on water, and/orthe like via hand gestures on objects, for example, scrap materialspicked up off the street.

Objects and/or toposculpted manifolds may be borrowed and shared. Thus,in one implementation, a user may use a dinner plate to make a rocketengine, and then pass the plate and/or rocket engine design along toanother user, who may use the plate to make a pipe organ sculpture,share that with the first user, and/or the like.

In some implementations, bricology may include sharing of objects, e.g.,as props that can be extruded, spun, twisted, and/or moved throughspace, such as to make things using metatouch and meta-haptics.

FIG. 3 depicts another schematic example of the sensing and displayapparatus 1300 of FIG. 1E in one embodiment.

FIG. 3 illustrates an example of a software application (programmedinstructions to be included in the program 907 of FIG. 1B) which may beused to instruct the user, such as about food preparation in a user'shome, using the sensing and display apparatus 1300 of FIG. 1E. There isdepicted a shared computer-mediated reality (such as the firstaugmediated-reality space 1000 of FIG. 1A) on a kitchen countertop asthe surface 130 with a sink 341 and a cooktop 334 having burners 331.The secret subject matter 340 is only visible within the group ofcollaborating users of the digital eye glass 180 and the digital eyeglass 190, such as may be reflected in access rights, privileges, roles,and/or the like data stored in a database. The public subject matter 240is visible to non-participants (persons not involved or not wearing thedigital eye glass 180).

Food items 330 on the countertop of the surface 130 may be sensed by thethree-dimensional camera system of the digital eye glass 180, and thedigital eye glass 190. Data is communicated between the processingapparatus 908 (of FIG. 1A) associated with each of the digital eye glass180 and the digital eye glass 190, and each processing apparatus 908 isconfigured to compute a more accurate three-dimensional model of thefood items in the first augmediated-reality space 1000.

FIG. 4 depicts another schematic example of the sensing and displayapparatus 1300 of FIG. 1A and/or of FIG. 1E, in one embodiment.

FIG. 4 illustrates an example of application of the sensing and displayapparatus 1300 to various types of activities that may be facilitated inthe first augmediated-reality space 1000 of FIG. 1A, such as health,wellness, personal and group fitness, exercise, entertainment, gaming,music, dance activities, and/or the like all of which may use thesensing and display apparatus 1300 of FIG. 1A and/or of FIG. 1E.

There is depicted a floor having and/or providing a surface 130. Thesurface 130 may be a desk or a tabletop, and/or the like. A texture ispositioned on the floor. The texture may, in one implementation, includea pseudorandom pattern, a pattern that may be deliberately made frompseudorandom tiles, naturally occurring by way of varied texture, dirt,scuff marks, the natural texture of wood grain of a hardwood floor, thenatural texture of stone, concrete, marble, rubber and/or carpeting,and/or the like. The texture may, in one implementation, form a pattern430. The homography of pattern 430 may, through algebraic projectivegeometry, form a group action under video-orbit software (e.g., includedwith the program 907 of FIG. 1A) to be executed by the processingapparatus 908 of FIG. 1A, responsive to an output of a three-dimensionalcamera image provided by the infrared transmitter 186 (which is anexample of the first sensory-phenomenon effector 912 of FIG. 1B) and bythe infrared receiver 185 (which is an example of the firstsensory-phenomenon sensor 910 of FIG. 1B). An example of video-orbitstabilization in one implementation is provided above in reference toFIG. 2B.

In one implementation, the group of coordinate transformations of thefloor texture falls in an orbit given by FORMULA {2}.

$\begin{matrix}{{f(x)} = \frac{{Ax} + b}{{c\; \dagger \; x} + d}} & {{FORMULA}\mspace{14mu} \{ 2 \}}\end{matrix}$

[A] is a 2 by 2 (2×2) linear operator, [b] is a 2 by 1 (2×1)translation, [c] is a 2 by 1 (2×1) chirp (projective), [d] is a scalarconstant, and [x] is a 2 by 1 (2×1) spatial coordinate, and [x]=[x1,x2]^(T).

Since the floor remains in this orbit, as viewed by a plurality ofwearers (e.g., users) of the sensing and display apparatus 1300, thesubject matter may be disambiguated from other subject matters not inthe same plane as the floor. In accordance with an option, thethree-dimensional sensing nature puts the floor itself in athree-dimensional model, and thus points located on thisthree-dimensional plane may be clustered and segmented. The secretsubject matter 340 may include, for example, song lyrics displayed onthe surface 130 of the floor, visible to one or more participants, suchas in an andantephonic walk. The andantephone is a musical instrumentthat is played in response to the user walking.

In accordance with an option, the sensing and display apparatus 1300 isused for physical (gymnastic) instruction (such as, yoga, dancing,acting, martial arts, running, sports, etc.) by way of an augmediatedreality (virtual three-dimensional) instructor generated by the program907 of FIG. 1A. In one implementation, the image of the virtualinstructor is projected into the first augmediated-reality space 1000 ofFIG. 1A by the first sensory-phenomenon effector 912 of FIG. 1B (forexample). If more than one user has their own instance of the sensingand display apparatus 1300, a selected instance of the sensing anddisplay apparatus 1300 may provide a projection of the virtualinstructor into the first augmediated-reality space 1000 of FIG. 1A;alternatively, the virtual instructor may be provided by an instance ofthe processing apparatus 908 that is not associated with any instance ofthe sensing and display apparatus 1300.

In some implementations where there are multiple instances of thesensing and display apparatus 1300 used, the body movements of the usermay be scanned by another instance of the sensing and display apparatus1300, and/or by an instance of the sensing and display apparatus 1300(or the first sensory-phenomenon effector 912 of FIG. 1B) that is fixedin a stationary position in the environment (the firstaugmediated-reality space 1000 of FIG. 1A). In some suchimplementations, the virtual instructor may assist and/or otherwiseinteract with the actions of the user(s).

In accordance with an option, the sensing and display apparatus 1300 isconfigured to provide a virtual mirror, e.g., full-length mirror,bathroom mirror, and/or the like (via an effector placed in the firstaugmediated-reality space 1000 in combination with programmedinstructions included with the program 907 of FIG. 1A), and isconfigurable to scan the user's body, e.g., for fitness trackingpurposes (weight gain, body measurements, and/or the like). Inaccordance with an option, the sensing and display apparatus 1300 may beconfigured to behave like the mirror through which a body scan may beperformed. In some implementations, the sensing and display apparatus1300 may be configured to passively scan, monitor, track, and/or thelike user fitness, body shape, body parameters, and/or the like, such ason a periodic, continuous, and/or triggered basis, and to providenotifications, alerts, recommendations, coupons, and/or the like basedthereon, such as by correlating measured body parameters with valuesassociated with notifications.

FIG. 5 depicts an example of a diagram indicating a timing andsequencing operation of the sensing and display apparatus 1300 of FIG.1E in one embodiment.

FIG. 5 illustrates an example of the timing and sequencing ofcollaborative spatial imaging provided by the program 907 (FIG. 1A) usedwith the sensing and display apparatus 1300 of FIG. 1E in oneembodiment. A sequencing of the comparametric and/or superposimetricprocess of the sensing and display apparatus 1300 is depicted in FIG. 5.The sensing and display apparatus 1300 may be configured, in oneimplementation, to emit a periodic pulse train of extramissivetransmissions (e.g., send and receive) such as infrared optical energy,as a pulse train signal 500. The first emission (e.g., from the infraredeffector mounted to the sensing and display apparatus 1300) provides aweaker illumination signal 510 (having a relatively weaker strength),followed by a medium illumination signal 520 (having relatively mediumstrength), and then followed by a stronger illumination signal 530(having a relatively stronger strength). The illumination signals arespaced apart in time, and emitted from the sensor one after the othertowards an object in the first augmediated-reality space 1000 of FIG.1A.

The program 907 of FIG. 1A may be configured, in one implementation, toinclude a synthetic aperture imaging constructor (e.g., as programmedinstructions) configured to be responsive to a lightspace generated byilluminators (e.g., the effectors) of a participant sharing the firstaugmediated-reality space 1000 of FIG. 1A. The program 907 may, in oneimplementation, be configured to include a comparametric compositor 404(depicted in one example in FIG. 1AA) and/or a superposimetriccompositor 405 (depicted in one example, in FIG. 1AA). Furtherdiscussion of comparametric and/or superposimetric compositorembodiments are provided above, such as in relation to FIG. 2B.

The medium illumination signal 520, having relatively medium strength,is a normal strength signal resulting in a normal exposure to thethree-dimensional camera (a sensor) of the digital eye glass 180 (or thesensing and display apparatus 1300). The weaker illumination signal 510is a weak signal, such as ¼ (25%) strength, resulting in athree-dimensional camera exposure, e.g., that is two f-stopsunderexposed. The stronger illumination signal 530 is a relativelystronger signal, such as one that is four times as strong as that of themedium illumination signal 520, resulting in a three-dimensional cameraexposure, e.g., that is two f-stops overexposed. The firstsensory-phenomenon effector 912 (FIG. 1B) may, in one implementation,send out (e.g., transmit or emit) an initial signal into the firstaugmediated-reality space 1000, and the first sensory-phenomenon sensor910 of FIG. 1B may receive the reflections from the objects located inthe first augmediated-reality space 1000 of FIG. 1B.

The result is three exposures or three “gettings” of information inwhich the first getting of the weaker illumination signal 510, is richin highlight details or details in areas of the getting that may orwould otherwise saturate the sensor. A getting is, in oneimplementation, a receiving of a reflected illumination, reflected froman object located in the first augmediated-reality space 1000 of FIG.1A, and/or of an original source illumination. The reflection from thestronger illumination signal 530 is rich in shadow detail or rich inweak-signal detail, for example, for more distant objects, or forobjects which have weak energy return (e.g., reflected back) to thethree-dimensional camera (e.g., the first sensory-phenomenon sensor 910of FIG. 1B).

The program 907 of FIG. 1B may be configured, in one implementation, tocombine these three gettings (the received reflected signals) in orderto determine more information from the scene (the firstaugmediated-reality space 1000) than any one getting might provideindividually. The program 907 may, in one implementation, be configuredto execute a comparametric analysis on the received (e.g., reflected)data in order to determine a result: e.g., an extended responsethree-dimensional camera may be provided to facilitate the comparametricanalysis by using a comparametric analysis program 406 (depicted in oneexample in FIG. 1AA and discussed in relation to FIG. 2B).

In another implementation, the sensitivity of the camera (sensor and/oreffector) may be adjusted to acquire three-dimensional images that are,e.g., weak, medium, and strong. Once these signals are received, theprogram 907 is configured to combine them, and then display the combinedimage to the user(s) via their instance of the second sensory-phenomenoneffector 914 of FIG. 1B. Additionally, visible light images may be soacquired.

In some implementations where there is a plurality of users present inthe first augmediated-reality space 1000 of FIG. 1B, the program 907 maybe configured to include a superposimetric analyzer 407 (depicted inFIG. 1AA and discussed in relation to FIG. 2B), and the program 907 maybe configured to determine a superposimetric spatial imaging, such as byusing a superposimetric spatial imaging program 408 (depicted in FIG.1AA and discussed in relation to FIG. 2B).

In one implementation, a first instance of the sensing and displayapparatus 1300 is configured to capture three-dimensional sceneinformation or images by way of a pulse train signal 500 (and by usingthe first sensory-phenomenon sensor 910 and the first sensory-phenomenoneffector 912 and the program 907 of FIG. 1B). A second instance of thesensing and display apparatus 1300 is configured to capturethree-dimensional information from the same scene (the firstaugmediated-reality space 1000 of FIG. 1B), using the pulse train signal509. The two instances of the sensing and display apparatus 1300 may, inone implementation, be configured to work in concert to spatially imagea three-dimensional scene, such as according to a timeline having timeslots, such as a first time slot 570, a second time slot 571, a thirdtime slot 572, a fourth time slot 573, a fifth time slot 574, and asixth time slot 575. Each time slot may be assigned to one or more usersof the first augmediated-reality space 1000 of FIG. 1A. For example, inthe illustrated implementation, the first time slot 570 may be assignedto the first user; the second time slot 571 to the second user; thethird time slot 572 to the third user; the fourth time slot 573 to thefourth user; the fifth time slot 574 to the first user; the sixth timeslot 575 to the second user, and so on. If more users are present in thefirst augmediated-reality space 1000, then more time slots may beallocated, e.g., according to the number of users in a sequential order.A non-sequential order may also be used, in some implementations.

For the first time slot 570, there are three divided time sectionsassigned to the first user of the first augmediated-reality space 1000of FIG. 1A. During the first time section, the digital eye glass 180captures a reflection of a weak signal (reflected from an object in thefirst augmediated-reality space 1000) due to the weaker illuminationsignal 510 transmitted from the digital eye glass 180 to a scene (thefirst augmediated-reality space 1000) shared with other users usingtheir own instance of the sensing and display apparatus 1300. During thesecond time section, the digital eye glass 180 captures a reflection ofa medium-strength signal due to the medium illumination signal 520transmitted from the digital eye glass 180 associated with the firstuser. During the third time section, the digital eye glass 180 capturesreflection of a relatively strong signal due to the strongerillumination signal 530 transmitted from the digital eye glass 180 (ofthe first user), and then subsequently reflected from a physical objectlocated in the first augmediated-reality space 1000 of FIG. 1B.

For the second time slot 571, there are three divided time sectionsassigned to the second user of the first augmediated-reality space 1000of FIG. 1B. During the first time section, the digital eye glass 190 (ofthe second user) captures a reflection of a weak signal due to a weakerillumination signal 511 transmitted from an effector (such as the firstsensory-phenomenon effector 912 of FIG. 1B) positioned on the digitaleye glass 190 and transmitted to a scene (first augmediated-realityspace 1000) shared by the users (each user involved has their owninstance of the sensing and display apparatus 1300). During the secondtime section, the digital eye glass 190 captures a reflection of amedium-strength signal due to a medium illumination signal 521transmitted from the digital eye glass 190 (of the second user). Duringthe third time section, the digital eye glass 190 captures a reflectionof a strong signal due to a stronger illumination signal 531 transmittedfrom the digital eye glass 190 (of the second user).

The same process is repeated for the third time slot 572 and the fourthtime slot 573 for the third user and the fourth user, respectively. Forthe fifth time slot 574, the cycle is repeated, back to the first user,and the weaker illumination signal 540, the medium illumination signal550, and the stronger illumination signal 560 are generated and used(similar to the first time slot 570). Then, the reflections are sensedby the sensors mounted on each instance of the sensing and displayapparatus 1300, and then are processed by each instance of the program907 associated with each (respective) instance of the sensing anddisplay apparatus 1300.

For the sixth time slot 575, the cycle is repeated, back to the seconduser, and the weaker illumination signal 541, the medium illuminationsignal 551, and the stronger illumination signal 561 are generated andused (similar to the second time slot 571), etc.

Any type of multiplexing and/or switching may be used in variousembodiments and implementations, such as time division multiplexing,frequency domain multiplexing, and/or the like.

In one embodiment, each instance of the sensing and display apparatus1300 is configured to implement three-dimensional HDR (High DynamicRange) imaging, such as by programmed instructions included in theprogram 907 of FIG. 1B, in which the program 907 includes acomparametric compositor, which may be implemented on a Wyckoff Set ofthree-dimensional images, scans, and/or the like. The comparametriccompositor is configured to combine a plurality of different gettings ofthe same subject matter by using a method 513 (see the example of FIG.5A).The method 513 may be provided, in one implementation, as executableprogrammed instructions in the program 907 of FIG. 1B.

FIG. 5A depicts an example of a method 513, to be included as programmedinstructions configured to direct the processing apparatus 908 (FIG. 1B)used in the sensing and display apparatus 1300 of FIG. 1B in oneembodiment.

The method 513 (of the comparametric compositor) includes operation 502,including, in one implementation, acquiring a Wyckoff Set. Oncecompleted, control is then passed over to operation 504.

Operation 504 includes, determining a comparametric relationship amongmembers of the Wyckoff Set acquired in operation 502. Once completed,control is then passed over to operation 506.

Operation 506 includes, determining a certainty function for each levelof each member of the Wyckoff Set. Once completed, control is thenpassed over to operation 508.

Operation 508 includes, constructing a weighted sum, for example byusing FORMULA {3}.

$\begin{matrix}{\hat{q} = \frac{\Sigma_{i}\mspace{14mu} c_{i}\frac{1}{k}q_{i}}{\Sigma_{i}\mspace{14mu} c_{i}}} & {{FORMULA}\mspace{14mu} \{ 3 \}}\end{matrix}$

In one implementation, [k] are the overall gains or amplitudes of eachgetting of the Wyckoff Set of gettings, [ci] are the certainties, and[q_(i)] are the photoquantigraphic measurement arrays (e.g., thethree-dimensional gettings themselves). Such a composition is called acomparametric compositor 409 (depicted in FIG. 1AA and discussed inrelation to FIG. 2B) and implemented in the method 513 of FIG. 5A.

Thus, even a single instance of the sensing and display apparatus 1300benefits from using the method 513. Data fusion (collection of data)from multiple instances of the sensing and display apparatus 1300 mayproceed, e.g., superposimetrically, by using a method 580 (see theexample of FIG. 5A). The method 580 may be configured as programmedinstructions to be included, in one implementation, in the program 907of FIG. 1B, and then to be executed by the processing apparatus 908 ofthe sensing and display apparatus 1300.

Operation 582 includes, acquiring a lightspace set of quantigraphs, [qi](e.g., a set of different quantigraphs, each quantigraph provided by adifferent illumination of the same subject matter). Once completed,control is then passed over to operation 584.

Operation 584 includes, determining a superposimetric relationship amongdifferent lightvectors of this set. Once completed, control is thenpassed over to operation 586.

Operation 586 includes, determining a certainty function for eachsuperposimetric law of composition. Once completed, control is thenpassed over to operation 588.

Operation 588 includes, constructing a weighted sum, such as inaccordance with the example in FORMULA {4}.

$\begin{matrix}{\hat{\overset{\sim}{q}} = \frac{\Sigma_{i}\mspace{14mu} c_{i}{\hat{q}}_{i}}{\Sigma_{i}\mspace{14mu} c_{i}}} & {{FORMULA}\mspace{14mu} \{ 4 \}}\end{matrix}$

In one implementation, [ci] are the overall gains or amplitudes of theHDR (high definition resolution) image composite due to each source ofillumination (i.e. each lightvector). Such a composition is called asuperposimetric compositor (implemented, for example, by the method 513of FIG. 5A).

The program 907 associated with a respective instance of the sensing anddisplay apparatus 1300 is configured, in one implementation, to generatea quantigraph information due to its own light source plus contributionsfrom the light sources in all the other instances of the sensing anddisplay apparatus 1300 involved with the first augmediated-reality space1000 of FIG. 1B.

The program 907 may be configured, in one implementation, to combine allof these lightspace quantigraphs (one lightspace graph from eachinstance of the sensing and display apparatus 1300) to form a masterlightspace quantigraph, such as to be shared among all the participatinginstances of the sensing and display apparatus 1300 in the firstaugmediated-reality space 1000 of FIG. 1B.

In one embodiment, the program 907 of FIG. 1B may include aspatial-imaging multiplexer 410 (such as depicted in one example in FIG.1AA) configured to multiplex a scanning from the digital eye glass 180of FIG. 1E, and the digital eye glass 190 of FIG. 1E.

For example, in one implementation, the spatial imaging multiplexer mayinclude a time-division multiplexer 411 (depicted in one example in FIG.1AA).

In one implementation, the time-division multiplexer (which may beconfigured as programmed instructions) may be configured to facilitatecooperation with instances of the sensing and display apparatus 1300worn by participants associated with the first augmediated-reality space1000 of FIG. 1B. For instance, in one implementation, a variableamplitude multiplexer is configured to: (A) cooperate with instances ofthe sensing and display apparatus 1300 worn by other participants in thefirst augmediated-reality space 1000; and (B) provide approximatelyidentical gettings of subject matter in response to different degrees ofillumination reflected from the subject matter.

FIG. 6 depicts an example of a real-time augmediated reality environmentshared among multiple participants using the sensing and displayapparatus 1300 of FIG. 1E in one embodiment.

FIG. 6 illustrates an implementation of collaborative spatial imagingand gesture-based interfacing with physical objects to enable real-timeaugmediated reality interfaces to be shared among multiple participantsusing the sensing and display apparatus 1300 of FIG. 1E. The sensing anddisplay apparatus 1300 may be configured, in one implementation, toinclude a collaborative gesture-based interface 412 (depicted in oneexample in FIG. 1AA) configured to allow multiple users to interact withreal-world three-dimensional physical objects (located in the firstaugmediated-reality space 1000 of FIG. 1B), such as a trackable box, orother instances of the object 600 located in the firstaugmediated-reality space 1000 of FIG. 1B.

The surfaces of the object 600 may, in one implementation, be annotatedwith markers, patterns, and/or three-dimensional geometric patterns, ortheir naturally occurring vertices, corners, edges, or surfaces may bedistinguishable by dirt, scuff marks, irregularities, textures, and/orthe like. These surfaces 601 are scanned by the digital eye glass 180(e.g., associated with a first user), the digital eye glass 190 (e.g.,associated with a second user), and the digital eye glass 196 (e.g.,associated with a third user). This facilitates shared tactileaugmediated reality applications (e.g., to be included in the program907, or to be included in the sensing and display apparatus 1300 or tobe included in a computer not associated with the sensing and displayapparatus 1300).

Users experience and/or interact with the first augmediated-realityspace 1000 (e.g., with any virtual objects and physical objects in thefirst augmediated-reality space 1000) of FIG. 1B through theirrespective instances of the sensing and display apparatus 1300, usinginteractive gestures while touching one or more real physical objectssuch as the object 600.

The digital eye glass 180, the digital eye glass 190, and the digitaleye glass 196 allow multiple users to interact with thesethree-dimensional physical instances of the object 600 (located in thefirst augmediated-reality space 1000 of FIG. 1B), such as a polyhedralobject, an object having a flat surface, an object having a recognizablesurface texture 602, an image icon 603, a feature recognizable withthree-dimensional structures (e.g., corners, vertices, edges), and/orsurfaces 601, in the first augmediated-reality space 1000 of FIG. 1Bthrough the sensing and display apparatus 1300. The program 907 of FIG.1B may be configured, in one embodiment, to execute the programmedinstructions configured for recognition of human gestures made in thefirst augmediated-reality space 1000 of FIG. 1B.

The gesture input 610, the gesture input 611, and the gesture input 612are recognized by instances of the digital eye glass 180, the digitaleye glass 190, and/or the digital eye glass 196, such with the use ofthree-dimensional detectors and/or sensors, range three-dimensionalsensors, and/or the like. A wireless communications module 630, awireless communications module 631, and a wireless communications module632 are configured to enable the flow of information, such as a gestureinput 610, a gesture input 611, and a gesture input 612 to betransmitted, shared, and to provide and/or facilitate user interactionbetween the users (or between objects) via their instance of the sensingand display apparatus 1300. The wireless communications module 631 maybe configured, in various implementations, for communication via avariety of wireless protocols, such as WiFi, Bluetooth, cellular, CDMA,HSPA, HSDPA, GSM, and/or the like. In one implementation, the pointinggesture is used to draw or annotate information or select indicia on ashared space or shared surfaces, to thus allow interaction amongmultiple wearers of the digital eye glass 180, the digital eye glass190, and the digital eye glass 196.

Gesture sensing may be facilitated, in one implementation, by using agesture-tracking detector. The gesture-tracking detector 413 is depictedin one example in FIG. 1AA.

The gesture-tracking detector 413 may, in one implementation, include aneural network 414 (depicted in one example in FIG. 1AA) configured toimplement a cost function, an estimator, and a gesture-tracking detector(which detector is responsive to an input from the estimator). To trainthe neural network, a cost function is first defined. This cost functionrepresents a logarithmic likelihood of a logistic regression.

In accordance with an option, the sensing and display apparatus 1300includes a best-fit optimizer 415 depicted in FIG. 1AA.

The best-fit optimizer is configured, in one implementation, to maximizea function for “best fit” through parameter selection. Minimizing thenegative of the function may be done, for example, through a gradientdescenter (e.g., negating the cost function), thus formulating thebest-fit optimizer in the context of a minimization problem by using thegradient descenter 416 (depicted in one example in FIG. 1AA).

The best-fit optimizer may be configured, for example. to maximize a LOG(logarithm) likelihood function. To prevent over fitting to the trainingdata, a regularizer 417 may be used in some implementations; theregularizer 417 is depicted in one example in FIG. 1AA.

The regularizer 417 implements a regularization term by adding thesquare of each parameter at the end of the cost function. Theseregularization terms will punish the cost function as the parametersbecome large. An overflow penalizer 418 may be used, and is depicted inone example in FIG. 1AA.

The overflow penalizer may be configured, in one implementation, tocompute a value that increases as the magnitude of any of the parametersincreases. An example of an overflow penalizer is an adder in whichthere is a magnituder on each input to the adder, so that the output isthe sum of the absolute values of the inputs. However, since theabsolute value function has a discontinuity in its first derivative, atthe origin, the overflow penalizer is an adder with a magnitude squareron each input. A magnitude squarer multiplies each input by its complexconjugate (i.e. [y]=[x][x]) to give the magnitude squared of the input.The overflow penalizer adds the sum of the square magnitudes of eachparameter to the cost function. This prevents or discourages (reducesthe occurrence of) a floating point overflow.

The training cost function J (θ) is provided, in one implementation, bythe example of FORMULA {5}.

J(θ)=l(θ)+R(θ, λ)   FORMULA {5}:

The term [l(θ)] is the logistic regression for minimization, provided byFORMULA {6}.

$\begin{matrix}{{l(\theta)} = {{- \frac{1}{s}}{\sum\limits_{i = 1}^{s}\; {\sum\limits_{j = 1}^{c}\; \lbrack {{y_{j}^{(i)}{\log ( {h_{\theta}( x^{(i)} )} )}_{j}} + {( {1 - y_{j}^{(i)}} ){\log ( {1 - ( {h_{\theta}( x^{(i)} )} )_{j}} )}}} \rbrack}}}} & {{FORMULA}\mspace{14mu} \{ 6 \}}\end{matrix}$

In one implementation, [s] denotes the total number of training cases,and [c] denotes the total number of output gestures. The objective ofthis function is to add up the cost from each of the training cases.Thus, [i] is used to denote the current training cases that are beingused to calculate the cost. [hθ(x(i))] denotes the estimation resultingfrom forward propagation. After calculating the estimate from forwardpropagation, a logistic function may be used to rescale that numberbetween 0 and 1.

The term R(θ, λ) is the regularization term provided by FORMULA {7}.

$\begin{matrix}{{R( {\theta,\lambda} )} = {\frac{\lambda}{2s}\lbrack {{\sum\limits_{i = 1}^{n}\; {\sum\limits_{j = 1}^{p}\; ( \theta_{i,j}^{(1)} )^{2}}} + {\sum\limits_{i = 1}^{c}\; {\sum\limits_{j = 1}^{n}\; ( \theta_{i,j}^{(2)} )^{2}}}} \rbrack}} & {{FORMULA}\mspace{14mu} \{ 7 \}}\end{matrix}$

In one implementation, [n] denotes the total number of nodes in thehidden layer, and [p] denotes the total number of nodes in the inputlayer, which is the number of pixels found in each of the trainingimages. Training the neural network includes, in one implementation,collecting the training data, such as by using a spatial-imaging sensorsuch as a depth-sensing three-dimensional camera. For example, thesensing and display apparatus 1300 may include programmed instructions(to be included in the program 907) configured to train the sensing anddisplay apparatus 1300 to recognize human gestures, such as thefollowing: (A) the framing gesture (e.g., both hands form the corners indiagonal of each other, to frame a scene or object within or beyond thehands of the user), (B) a finger pointing gesture, and/or the like. Insome implementations, a human-gesture recognition program 419, gesturelibrary, and/or the like may be used (depicted in one example in FIG.1AA).

The program 907 may, in one implementation, be configured to facilitatedata collection. Collecting a large amount of training data is a way toimprove the performance of a learning method (programmed instructionsused in the program 907). In the setting (in the firstaugmediated-reality space 1000), sample data may be collected byrecording additional gesture samples, e.g., by wearing the sensing anddisplay apparatus 1300 (in daily usage) continuously (for an extendedperiod of time). High accuracy may be achieved by constantly streaminguser gestures and labeling the gesture with the correct identificationlabels, number, indicia, tags, ratings, and/or the like. This may, inone implementation, be effected in a social networking site, with acommunity of users each having their instance of the sensing and displayapparatus 1300. This continuous data collection approach results incontinued improvement to the learning software; the learning software424 (depicted in one example in FIG. 1AA) may be used.

In one implementation, in order to avoid over fitting the training data,fuse software (e.g., configured as programmed instructions) may beimplemented (included with the program 907). Analogous to a physicalfuse in an electric circuit which blows (stops conducting current) whentoo much current is consumed, the fuse software may be configured toblow (e.g., stop operations) when triggered, such as when too much timeis consumed for fitting, training, data collecting, and/or the like. Inone implementation, the fuse software may be implemented in accordancewith a method having operation (A), operation (B), and operation (C).

Operation (A) includes, splitting data between training data and testdata. For example, a separated 80% of the data is identified as trainingdata, and the remaining 20% is identified as test data. Once completed,control is passed on to operation (B).

Operation (B) includes, on every iteration of neural net training,running a forward propagation to acquire a gesture prediction accuracyand cost on both training and test sets. Control is passed on tooperation (C).

Operation (C) includes, considering the cost on both training and testsets along with the number of training iterations.

At some point (e.g., around 2000 iterations), the cost of the test datastarts to increase while the cost of the training data is stilldecreasing. This implies that after approximately 2000 iterations, theneural network is being over trained to the training data; that is, forthe case where the neural network is left to train forever, the neuralnetwork might only match items in the training data, and may reject mosteverything else.

In accordance with an option, the program 907 includes programmedinstructions for achieving, for the training stage of the neuralnetwork, an accuracy of approximately 99.8%. The cross-validation of thetrained neural network achieves an accuracy of approximately 96.8%. Forexample, the performance in frames-per-second (FPS) of only thegesture-recognition software may be approximately 100 FPS while theperformance of the sensing and display apparatus 1300, including gesturerecognition software, is approximately 30 FPS.

FIG. 7 depicts another example of a real-time augmediated reality sharedamong multiple participants (users) using the sensing and displayapparatus 1300 of FIG. 1E in one embodiment.

More specifically, FIG. 7 illustrates an example of collaborativespatial imaging and gesture-based interfacing with virtual or physicalobjects using the sensing and display apparatus 1300 of FIG. 1E. Thereis depicted an example of the digital eye glass 180 used with a virtualbubbles metaphor, in one implementation, in which a virtual object or areal object 700 is touched by a hand 710 of a user of the bubblemetaphor (a thing that acts like a bubble). A bubble is, in theillustrated example, a symbol of an icon, in which the icon appears as abubble, and behaves like a bubble (to some extent) to the user via theuser interface 800 of FIG. 8A. The program 907 may be configured, in oneimplementations, to include a bubble-metaphor generator 420 (depicted inone example in FIG. 1AA) in which, for example, images of bubbles (e.g.,three-dimensional) appear on the user interface 800 (to the user). Forexample, in one implementation, when a hand intersects the sphere of thebubble image, the bubble image changes to a motion picture (e.g., videoimage), animated graphic, interactive animation (e.g., such as may besensitive to the position of user appendages and/or objects in theenvironment), and/or the like of a bubble bursting (also shown in theexamples of FIG. 8B and FIG. 8C); then, the bursting bubble 720disappears from the screen (the user interface 800). In someimplementations, a bubble is an example of a menu item or a menu icon.

The bubble-metaphor generator 420 may, in one implementation, beconfigured to define a sphere and the volumetric contents of volume ofthe sphere by using FORMULA {8}.

4/3πr³   FORMULA {8}:

In one implementation, [r] is the radius of the sphere (the bubble). Thevolume of a sphere is zero since its surface exists on a set measure ofzero in the three-dimensional space.

However, the volume contained inside a sphere is [4πr 3] and this volumetherefore has a non-zero measure in three-dimensional space. Similarly,a point cloud of data from the hand 710 of the user is calculated andwhen the hand 710 and/or its point cloud intrudes upon the volume and/orpoint cloud of the sphere, an interference signal is generated by theprogram 907.

The program 907 may, in one implementation, be configured to execute anoperation, including calculating the strength of the interferencesignal, such as according to a Neyman-Pearson constant false alarm rate,and statistically thresholded to burst the bubble when the intrusion issufficient. In one implementation, a statistical significance test maybe performed upon the degree of point cloud intrusion of an appendage ofthe user (fingers, hand, legs, feet, and/or the like) into the volume ofthe sphere (the bubble).

The program 907 may be configured, in one implementation, to execute anoperation including, performing such measure of the degree of intrusioninto a sphere (or other shape); the programmed instructions forexecuting this operation may be referred to as a spherical volumetricintrusion estimator 421 (depicted in one example in FIG. 1AA); when suchan estimator is thresholded or used as a trigger for other actions, theprogrammed instructions may be referred to as a volume-intrusiondetector 422 (depicted in FIG. 1AA). The volume-intrusion detector 422is configured to detect the intrusion of a spherical volume (the bubble)and/or other volume of any shape.

The bubble-metaphor generator 420 includes programmed instructions to beincluded, in one embodiment, in the program 907, and is configured, inone implementation, to create a video image, animated graphic,interactive animation, and/or the like of one or more bubbles, balloons,balls, or other two-dimensional or three-dimensional shapes and/oricons, and causes each shape to vanish (or self-destruct, or burst) whenthe point cloud of all or a part of a user's body intersects to asufficient and/or specified degree with the interior area or volume ofthe two-dimensional or three-dimensional shape.

In accordance with an option, a bubble-bursting program 423 (depicted inone example in FIG. 1AA, and may be called a bubble-bursting metaphor)may include one or more representations of physical objects, e.g.,balls, that may be used, and when touched by the user, are interactedupon to become bubbles (e.g., virtual icons). Using physical objects(real objects in the first augmediated-reality space 1000 of FIG. 1A)may add an element of taction to the bubble-metaphor generator asdescribed previously for the table top, floor surface and/or the like.

The sensor (e.g., receiver) of both instances of the sensing and displayapparatus 1300 may, in one implementation, receive an image illuminatedby the effectors of the instances of the sensing and display apparatus1300, so in the case of two instances of the sensing and displayapparatus 1300, each instance of the sensing and display apparatus 1300captures six gettings: three gettings from the light source of thesensing and display apparatus 1300, and three gettings as the scene (thefirst augmediated-reality space 1000) appears illuminated by effectorsassociated with the other instance of the sensing and display apparatus1300. These sets of differently exposed and differently illuminatedimages provide additional spatial information, though not merelyphotometric stereo, but also, in some implementations, athree-dimensional spatial imaging and sensor fusion.

FIG. 8A depicts a schematic example of a user interface 800 for use withthe sensing and display apparatus 1300 of FIG. 1E in one embodiment.

In some implementations, the user interface 800 includes (and is notlimited to) a first interface section configured to display phenomenaderived from a first augmediated-reality space 1000 via the interfaceassembly 902 (FIG. 1A) configured to convey sensor signals 904 andeffector signals 906 with the first augmediated-reality space 1000. Theuser interface 800 also includes a second interface section configuredto display phenomena derived from a second augmediated-reality space1002 via the interface assembly 902 configured to face, detect, interactwith, and/or the like a second augmediated-reality space 1002. Thesecond interface module 905 is configured to convey sensor signals 904and effector signals 906 with the second augmediated-reality space 1002.The effector signals 906 are user presentable, at least in part, in anyone of the first augmediated-reality space 1000 and the secondaugmediated-reality space 1002.

The user interface 800 is provided, in one implementation, by theprogram 907 of FIG. 1A. FIG. 8A depicts a schematic example of the userinterface 800 using bubbles (as icons) for a first set-up operation. Insome implementations, such as to attain aesthetic minimalism andphysical safety, once the finger enters the field of view, the bubblesrise toward the field of view 802 in the user interface 800. The fieldof view 802 is associated with an effector of the sensing and displayapparatus 1300 projecting the user interface 800 to the eye of the userof the sensing and display apparatus 1300. The user interface 800 is foruse by a user 801. The user interface 800 is viewable through thesensing and display apparatus 1300. In one implementation, the field ofview 802 within the user interface 800 may include a grid; the gridincludes one or more cells, such as may be arranged as a matrix (e.g.,as in a spreadsheet). In the example of FIG. 8A, the cells are shown asrectangles that are identical in size and laid out in a regularrectangular grid of non-overlapping cells. However, in otherimplementations, the grid and/or its cells may be of different sizes,shapes, orientations, and/or the like, may overlap, may be laid in atriangular lattice, honeycomb arrangement, and/or any other regular orirregular pattern. The user interface 800 includes one or more bubblesand/or a bubble set 803 (or sets), such as may be arranged and/orcollected in one or more groups or sets. By way of example, there aredepicted four instances of the bubble set 803, including a work bubble804, a media bubble 805, a play bubble 806, and a social bubble 807,and/or the like. In one implementation, each bubble of the bubble set803, once selected, allows the user to view a setting, application,collection of data, and/or the like associated with the selected bubble.

FIG. 8B depicts a schematic example of a user interface 800 for use withthe sensing and display apparatus 1300 of FIG. 1E in one embodiment.FIG. 8B depicts a schematic example of the user interface 800 usingbubbles (as icons) for a second set-up operation. An application foldermay be opened, in one implementation, by popping the icon associatedwith the application folder, such as may be detected viagesture-recognition techniques (e.g., programed instructions to be usedin the program 907); once the application folder is opened, the otherbubbles blow out of the field of view 802. In one implementation, anapplication folder includes a collection of applications and/orselectable icons, shortcuts, bubbles, and/or the like associatedtherewith.

FIG. 8C depicts a schematic example of a user interface for use with thesensing and display apparatus 1300 of FIG. 1E in one embodiment. FIG. 8Cdepicts a schematic example of the user interface 800 using bubbles (asicons) for a third set-up operation. An application folder is opened, inone implementation, by popping the application folder, such as may bedetected using gesture-recognition techniques; once the applicationfolder is opened, the other bubbles blow out of the field of view 802.

FIG. 9A depicts a schematic example of the user interface 800 for usewith the sensing and display apparatus 1300 of FIG. 1E in oneembodiment. FIG. 9A depicts a schematic example of the folder selectionin the user interface 800. Folder icons may, in one implementation,spiral vertically about the imaginary x-axis of the field of view 802.Other forms of animation may be employed in other implementations.

FIG. 9B depicts a schematic example of the user interface 800 for usewith the sensing and display apparatus 1300 of FIG. 1E in oneembodiment. FIG. 9B depicts a schematic example of the folder selectionin the user interface 800. The folders are now in view, and ready forfinger selection.

FIG. 10A depicts a schematic example of a user interface 800 for usewith the sensing and display apparatus 1300 of FIG. 1E in oneembodiment. FIG. 10A depicts the schematic example of the applicationselection in the user interface 800. Applications are selected andentered in the field of view 802 similarly to the folders. In oneimplementation, a ratio of 1 to 5 (approximately) may be used for thethree-dimensional opaque logo to bubble ratio, for icons of the softwareapplications.

FIG. 10B depicts a schematic example of a user interface 800 for usewith the sensing and display apparatus 1300 of FIG. 1E in oneembodiment. FIG. 10B depicts a schematic example of applicationselection in the user interface 800. Logos of selected applications may,in one implementation, spiral, rotate, and/or the like rapidly to a topcorner of the user interface 800. In one implementation, a ratio of 1 to10 (approximately) is used for sizing the logo to a cell (e.g., one thatis part of a 3x3 grid of cells).

FIG. 11A depicts a schematic example of a user interface 800 for usewith the sensing and display apparatus 1300 of FIG. 1E in oneembodiment. FIG. 11A depicts a schematic example of the settingsselection used in the user interface 800. Once the finger (of the user801) is tracked on the top right section of the field of view 802, asettings bubble, button, and/or the like 808 emerges in oneimplementation. This may occur, for example, both within an application(for local settings) and from the home screen (for global settings). Theuser interface 800 thus, in one implementation, includes a settingsbubble 808.

FIG. 11B depicts a schematic example of a user interface for use withthe sensing and display apparatus 1300 of FIG. 1E in one embodiment.FIG. 11B depicts a schematic example of the level selection in the userinterface 800. In one implementation, once settings are selected, thesetting may spiral inwards about the rightmost column of the grid. Thesettings bubble 808 may include, connect to, and/or provide access to,for example: a setting type 810 (for setting up brightness), a settingtype 811 (for setting up loudness), a setting type 812 (for setting up anetwork connection), a setting type 813 (for setting up a meditationmode), and/or the like.

FIG. 11C depicts a schematic example of a user interface 800 for usewith the sensing and display apparatus 1300 of FIG. 1E in oneembodiment. FIG. 11C depicts a schematic example of a setting selection(in the user interface 800) for gummy settings; for example, once thebrightness setting is selected, a gummy line is stretched from thecenter of the brightness setting icon to the finger of the user. In oneimplementation, the scale controls the magnitude.

FIG. 12 depicts a schematic example of a user interface 800 for use withthe sensing and display apparatus 1300 of FIG. 1E in one embodiment.FIG. 12 depicts a schematic example of a meta view (a bird's eye view)of a visual search menu for the user interface 800.

FIG. 13 depicts a schematic example of the sensing and display apparatus1300 (also called a hardware array) of FIG. 1E in one embodiment. FIG.13 depicts a schematic example of the hardware used in the sensing anddisplay apparatus 1300 in one implementation. The sensing and displayapparatus 1300 may, for example, include a first phenomenon interface1302 configured to interface with a first augmediated-reality space1000, and a second phenomenon interface 1306 configured to interfacewith a second augmediated-reality space 1002. An example of the firstphenomenon interface 1302 includes the interface assembly 902 of FIG. 1Aand/or FIG. 1B.

In accordance with an option, the first phenomenon interface 1302includes a first sensory phenomenon effector (such as the firstsensory-phenomenon effector 912 of FIG. 1B) and a first sensoryphenomenon sensor (such as the first sensory-phenomenon sensor 910 ofFIG. 1B) each configured to operatively interact with the firstaugmediated-reality space 1000. The second phenomenon interface 1306includes a second sensory phenomenon effector (such as the secondsensory-phenomenon effector 914 of FIG. 1B) and a second sensoryphenomenon sensor (such as the second sensory-phenomenon sensor 916 ofFIG. 1B) each configured to operatively interact with the secondaugmediated-reality space 1002. An example of the first sensoryphenomenon effector includes the first sensory-phenomenon effector 912of FIG. 1B, and an example of the first sensory phenomenon sensorincludes the first sensory-phenomenon sensor 910 of FIG. 1B. An exampleof the second sensory phenomenon effector includes the secondsensory-phenomenon effector 914 of FIG. 1B, and an example of the secondsensory phenomenon sensor includes the second sensory-phenomenon sensor916 of FIG. 1B.

In accordance with an option, a processing assembly 1304 is configuredto operatively couple to the first phenomenon interface 1302 and thesecond phenomenon interface 1306. The processing assembly 1304 isconfigured to operatively interact with the first sensory phenomenoneffector, the first sensory phenomenon sensor, the second sensoryphenomenon effector, and the second sensory phenomenon sensor. Anexample of the processing assembly 1304 includes the processingapparatus 908 of FIG. 1A and/or FIG. 1B. and may, in one implementation,be configured to operatively couple to the first phenomenon interface1302 and the second phenomenon interface 1306.

The sensing and display apparatus 1300 includes (for example) the firstphenomenon interface 1302. The first phenomenon interface 1302 mayinclude a spatial-imaging sensor configured to include a finger tracker,a depth camera, and/or the like. The first phenomenon interface 1302 maybe an example of a combination of the first sensory-phenomenon sensor910 and/or the first sensory-phenomenon effector 912 of FIG. 1B. Thefirst phenomenon interface 1302 is configured, in some implementations,to detect, monitor and/or track the wearer's (e.g., user's) appendages(hands, fingers, arms, legs, feet, and/or the like). Examples of thefirst phenomenon interface 1302 includes a camera, a three-dimensionalspatial sensor, a three-dimensional scanning device, a three-dimensionalsensing device, and/or the like. The processing apparatus 908 may beconfigured to execute spatial-imaging software. The apparatus 1300further includes a second phenomenon interface 1306 configured to bewearable. The second phenomenon interface 1306 may be configured to be ahead-mounted display, a see-through head-mounted display, a binocularhead-mounted display, and/or the like.

The apparatus 1300 also, in one embodiment, includes the processingassembly 1304, which, in one implementation, is an example and/orcomponent of the processing apparatus 908 of FIG. 1A, and may include acentral processing unit and/or a graphics-processing unit, and/or thelike.

The apparatus 1300 also includes, in one embodiment, the secondphenomenon interface 1306, such as a spatial-imaging display that mayinclude a head mounted display unit (to form a binocular opticalsee-through head mounted display, for example). In one implementation,the second phenomenon interface 1306 is configured to permit the viewerto view the first augmediated-reality space 1000 and to view the secondaugmediated-reality space 1002 of FIG. 1A (e.g., at the same time). Inone implementation, a depth map 1308 is provided by the first phenomenoninterface 1302 to the processing assembly 1304 (and/or to the program907 of FIG. 1A). The second phenomenon interface 1306 is an example of acombination of the second sensory-phenomenon sensor 916 and/or thesecond sensory-phenomenon effector 914 of FIG. 1B.

The depth map 1308 is, in one implementation, a digital map of thephysical artifacts identified in the scene (e.g., in the firstaugmediated-reality space 1000 of FIG. 1A) by the first phenomenoninterface 1302.

FIG. 14 depicts a schematic example of the apparatus 1300 of FIG. 13 inone embodiment. The apparatus 1300 provides a combination of componentsconfigured to facilitate augmediated reality functionality. Theapparatus 1300 may be implemented in a personal computer (such as, forexample, a desktop, a laptop, a tablet, or a smart phone, and/or thelike). In one implementation, the apparatus 1300 may be implemented as adevice that may be worn on the head of the user as a pair of eyeglasses. In one implementation, the apparatus 1300 includes hardwarecomponents configured to provide the user interface 800 of FIG. 8A, anda set of software applications (e.g., programmed instructions in theprogram 907 of FIG. 1A).

In one implementation, the apparatus 1300 is configured to facilitateviewing in a stereoscopic three-dimensional image on the secondphenomenon interface 1306. The apparatus 1300 is configured to operatethrough and/or recognize all or any subset of user-gestural movements inthe augmediated reality environment (e.g., in the firstaugmediated-reality space 1000 of FIG. 1A). The apparatus 1300 may, inone implementation, be configured to recognize controlled movements(gestures) of the user's body and/or appendages (e.g., fingers, hands,arms, legs, feet, and/or the like), such as to simulate interaction withcomputerized projections (such as, for example, a virtual keyboard) thatare projected by an effector device. The effector device may projectimages in two-dimensions and/or three-dimensions that may be viewed inone or more coordinate systems. Such coordinate systems may be, forexample, anchored to either the second phenomenon interface 1306 or thebody, or registered to real world objects in the firstaugmediated-reality space 1000.

In addition, computerized applications (programmed instructions such as,productivity solutions, games, media, and/or the like) may, in oneimplementation, be controlled in a three-dimensional augmediated reality(e.g., in the first augmediated-reality space 1000 or the secondaugmediated-reality space 1002) through the apparatus 1300, such as withuse of the user's finger, hand and/or arm movements in the space infront of the second phenomenon interface 1306, and may be registered inrelation to physical components such as, for example, surfaces,window-sills and doorframes.

The apparatus 1300 can be configured for a variety of uses, such as, forexample: gymnastic instruction, such as dancing, martial arts, trackrunning, sports instruction, and/or the like in which an augmediatedreality (e.g., virtual three-dimensional) instructor demonstratesmovement. With two or more users wearing the apparatus 1300, theaugmediated reality instructor is configured to correct the movement ofthe students (users) through cross-recognition of the movement throughthe apparatus 1300. The apparatus 1300 can be configured for a varietyof uses, such as, for example: viewing full-stereoscopicthree-dimensional movies that are fixed relative to a user's head, or toa surface. The apparatus 1300 can be configured for a variety of uses,such as, for example: an augmediated reality keyboard, where the trackedfinger swipes through the keys and constructs words using logic,programmed instructions, and/or the like, such as those described belowin SOFTWARE APPLICATIONS. Further examples are described below inSOFTWARE APPLICATIONS. In some implementations, the components ofapparatus 1300 may be reduced, miniaturized, and/or the like, such as tofit, for example, on a pair of eye-glass frames in which the componentsof the apparatus 1300 may be imbedded in the frame of the eye-glassframe. The apparatus 1300 includes a cooperative combination of hardwaredevices and software (programmed instructions) configured to control theuser interface 800 (FIG. 8A) and a set of software applications.

Referring to FIG. 13, in accordance with an option, the apparatus 1300includes a combination of a first phenomenon interface 1302, a secondphenomenon interface 1306, and a processing assembly 1304.

In accordance with another option, the apparatus 1300 includes theprocessing assembly 1304 operatively coupled to a first interface module903 configured to operatively couple to the first phenomenon interface1302, and also operatively coupled to a second interface module 905configured to operatively couple to the second phenomenon interface1306.

In accordance with yet another option, the apparatus 1300 includes aninterface system. The interface system includes the first interfacemodule 903 configured to operatively couple to the first phenomenoninterface 1302, the second interface module 905 configured tooperatively couple to the second phenomenon interface 1306, and a thirdinterface configured to operatively couple to the processing assembly1304.

In one implementation, the apparatus 1300 includes a first phenomenoninterface 1302 configured to detect sensory phenomena (such as, visualphenomena, audio phenomena, tactile phenomena, and/or the like) receivedfrom the first augmediated-reality space 1000 of FIG. 1A. The firstphenomenon interface 1302 may also, in one implementation, be configuredto provide and/or transmit sensory phenomena (such as, an image, etc.)to the first augmediated-reality space 1000 of FIG. 1A. In oneimplementation, the first phenomenon interface 1302 may also beconfigured to be oriented (pointed) away from the eye of the user andtoward the first augmediated-reality space 1000.

In one implementation, the first phenomenon interface 1302 includes afirst phenomena sensor and a first phenomena effector. Reference to afirst phenomena sensor may include one or more single instances (one) ofthe first phenomena sensor. It will be appreciated that reference to afirst phenomena effector may include one or more single instances (one)of the first phenomena effector. The first phenomena sensor may, forexample, include a spatial-imaging sensor, a depth camera, afinger-tracking camera (including respective software), and/or the like.Examples of the depth camera include: (i) the Asus™ Xtion Pro™ system;(ii) the Leap-Motion™ system; (iii) the Creative™ Interactive GestureCamera; (iv) the PMD™ CamBoard nano, and/or the like. The firstphenomena effector may include a speaker, a lamp, an illuminationdevice, a video or holographic projector, a laser, and/or the like.

In one implementation, the first phenomenon interface 1302 (e.g., by wayof the first phenomena sensor) is configured to calculate, generate,and/or provide a depth map 1308 to the processing assembly 1304. Thefirst phenomenon interface 1302 is configured to provide (return)coordinates and spatial orientation of various physical components (suchas surfaces, windows, doorways) located in the first augmediated-realityspace 1000 of FIG. 1A. The depth map 1308 is configured to allowtracking of the physical components (objects), such as for registrationpurposes. The first phenomenon interface 1302 may, in oneimplementation, include a finger-tracking depth camera configured toprovide the depth map 1308. The first phenomenon interface 1302 isconfigured to transfer or transmit the depth map 1308 to a memoryassembly 909 (FIG. 1A) coupled to the processing assembly 1304. Software(programmed instructions) stored in the memory module is configured touse the depth map 1308, such as to track the appendages of the user(finger, hand, arm, leg, foot) and/or physical component is located inthe first augmediated-reality space 1000 (such as, a door frame, awindow, a surface, ad/or the like). The depth map 1308 may be accessedand/or used by other software applications (programmed instructions)that are used with the apparatus 1300. The software stored in the memoryassembly 909 may, in one implementation, be configured to calculate thephysical [X,Y,Z] coordinates of the tracked item based on the depth map1308 provided by the camera. The [X,Y,Z] mapping (depth or dimensionalmapping) may occur within a coordinate system associated to the body ofthe user and/or the apparatus 1300 itself, or to a registered plane(e.g., associated with a physical object), for example.

For some implementations where the user wears the apparatus 1300, thefirst phenomenon interface 1302 faces and/or is otherwise oriented in adirection of a field of view of the user (also called the firstaugmediated-reality space 1000 of FIG. 1A). In one implementation, thefield of view of the user is an eye forward viewing direction. Forexample, in one implementation, the field of view of the user may be adirection in which the user may be able to view the location of theuser's fingers. For the case where the user is typing on a virtualkeyboard, the user's fingers may be tracked by the first phenomenoninterface 1302, even if the user is looking elsewhere. In oneimplementation, the second phenomenon interface 1306 faces a directiontoward (e.g., is oriented toward) the eyes of the user (the secondaugmediated-reality space 1002 of FIG. 1A).

By way of example, the first augmediated-reality space 1000 may be,represent, and/or be associated to a physical place in which at leasttwo users having their own instance of the apparatus 1300 may interactwithin the first augmediated-reality space 1000 (e.g., collaborate,compete, fight, and/or the like), by user interaction with the sensoryphenomena (within the first augmediated-reality space 1000) provided byat least one or more instances of the apparatus 1300 associated with atleast one or more users.

By way of example, the first augmediated-reality space 1000 may be,represent and/or be associated with a physical place in which at leastone user has an instance of the apparatus 1300 and may interact withinthe first augmediated-reality space 1000 by user interaction with thesensory phenomena (within the first augmediated-reality space 1000)provided by the first phenomenon interface 1302 of the apparatus 1300.

The second phenomenon interface 1306 is configured, in oneimplementation, to detect sensory phenomena received from a user (suchas tracking eye movements, facial expressions, and/or the like). Thesecond phenomenon interface 1306 may also be configured to provideand/or transmit sensory phenomena (such as an image, video, graphics,sound, and/or the like) to the user in a second augmediated-realityspace 1002 of FIG. 1A. The second phenomenon interface 1306 may also, inone implementation, be configured to be oriented (e.g., pointed) towardan eye or both eyes of the user and away from the firstaugmediated-reality space 1000. In other implementations, the secondphenomenon interface 1306 may be configured to be oriented toward anarea that is behind, above, below, around, remote, and/or the like withrespect to the user.

In one implementation, the second phenomenon interface 1306 includes asecond phenomena sensor and a second phenomena effector. Reference to asecond phenomena sensor may include one or more single instances (one)of the second phenomena sensor. Reference to a second phenomena effectormay include one or more single instances (one) of the second phenomenaeffector. The second phenomena sensor may, in various implementations,include a spatial-imaging sensor, a depth camera, a finger-trackingcamera (including respective software), and/or the like. The secondphenomena effector may, in one implementation, include an audio output(e.g., earpiece), and/or a display unit configured to display images tothe eye of the user.

The processing assembly 1304 may be referred to in some instances as aspatial-imaging processing assembly, and is an example of the processingapparatus 908 of FIG. 1A. Examples of the processing assembly include acentral processing unit, a graphics processing unit, microprocessor,application-specific integrated circuit (ASIC)and/or the like. Theprocessing assembly 1304 may be configured, in one implementation, tocompute spatial positions and orientations of objects, such as of theskeleton of fingers, hands or arms, and/or other appendages of the user,based on the information provided by and/or received from the firstphenomenon interface 1302. The processing assembly 1304 is configured toexecute programmed instructions, and may, in some implementations, beconfigured with and/or coupled to networking connectivity (wireless orwired). In one implementation, the processing assembly 1304 may beconfigured to provide a signal representing sensory phenomena (e.g.,spatial images) to the second phenomenon interface 1306 (in the secondaugmediated-reality space 1002 of FIG. 1A). The processing assembly 1304may include for example: a mobile phone, a mobile computer (e.g.,attached to and/or built into a belt), a stationary computer (e.g.,placed on a desk), and/or the like.

The processing assembly 1304 is configured to execute software(programmed instructions) and configured, in one embodiment, tocalculate spatial coordinates of positions and orientations of fingers,hands, and/or arms (of the user and/or of the users in the firstaugmediated-reality space 1000), such as based on the informationincluded in the depth map 1308.

The apparatus 1300 may, in one implementation, include a user-wearableframe configured to be worn by the user (such as a head-mountable frame,eye-glass frame, visor, mask, and/or the like), and may also beconfigured to support any combination of the components of the apparatus1300.

Software Applications

The software applications (programmed instructions), described below,may be configured to cooperate with the apparatus 1300. In oneimplementation, the software applications are configured to use thedepth map 1308 (FIG. 13) stored in the memory assembly 909 of FIG. 1A.The depth map 1308 is provided by the first phenomenon interface 1302(e.g., via a sensor associated therewith), such as to the processingassembly 1304. Various software applications may be configured toproduce graphical images based on the depth map 1308, and to display thegraphical images to the user (e.g., via the second phenomenon interface1306). In one implementation, the graphical images depict correspondingmovements of a tracked finger, hand, arm, leg, foot, wand, and/or otheruser appendage and/or physical component (e.g., window, door, surface,and/or the like) located in the first augmediated-reality space 1000.

Examples of the software applications include a user-interfaceapplication configured to manage the user interface 800 (depicted inFIG. 8A) for a shooting game. The software applications are configuredto facilitate user interaction techniques and user tracking (ofgestures); for example, in one implementation, the index finger of theuser may be tracked as a barrel of a gun, and the shortening of thedistance between the thumb end and the palm (by bending the thumb),indicates the user's desire to trigger the firing of a virtual gun. Thesoftware applications are configured to augment images, such as virtualtargets displayed in the first augmediated-reality space 1000 of FIG.1A, in which the targets may be shot at with the virtual gun, and theimages are displayed to the user, via the second phenomenon interface1306.

An example of the software application includes an augmediated realityshooting game. The software application is configured to facilitate userinteraction techniques and user tracking (e.g., in a similar way as thatused in the shooting game). The software application is configured todisplay a virtual target that is registered relative to the realphysical world and/or objects therein. The virtual target is shot atusing a virtual gun or weapon. The registration (between the virtualtarget and the physical object) may occur by the tracking of thephysical components (e.g., room components), such as by way of the depthmap 1308.

An example of a software application may include an augmediated realitykeyboard. The software application is configured to provide and/ordisplay graphics to the user: a virtual keyboard exists underneath thefield of view shown by the second phenomenon interface 1306 to the user.Once the interaction technique (e.g., user gesturing motion) is detectedby a sensor of the apparatus 1300, such as “Open Hand Raising/LoweringKeyboard” gesture, the software is configured to cause an image of thekeyboard to rise and enter the field of view of the user. In oneimplementation, the virtual keyboard may be fixed within a coordinatesystem of the second phenomenon interface 1306, of the user's body,relative to a registered physical component, relative to an orientationof the apparatus 1300, and/or the like. In one implementation, thesoftware is configured to interact with the user in the followingmanner. A user gesture is used by the user for an Open HandRaising/Lowering Keyboard gesture. Once an opened hand is visible (e.g.,at least a threshold number of finger tips, e.g., two) for more than apredetermined amount of time (e.g., one second), the software isconfigured to raise the augmediated reality keyboard to the field ofview of the user. The user may initiate a “Swiping Finger Input Device”gesture that the software is configured to respond thereto. The softwareis configured to detect a gesture in which the open hand gesture may bedisabled (e.g., by closing the hand, or tracking just one finger), andin response, the software causes the image of the augmediated realitykeyboard to be lowered to remove the keyboard from the field of view ofthe user. The lowering of the image of the augmediated reality keyboardmay be accomplished once the hand (of the user) is open again for morethan a predetermined amount of time (e.g., one second).

In one implementation, the software may be configured to respond toanother user interaction technique, including a swiping finger inputdevice. The user swipes through the desired keys of the augmediatedreality keyboard. This may be achieved, in one implementation, by fixingboth the tracked finger's motion as well as the static location of thekeys to the same z plane. In this example configuration, the [X,Y,Z]coordinate system may be fixed to the second phenomenon interface 1306,the user's body, to an orientation of the apparatus 1300, to aregistered physical component in the first augmediated-reality space1000 such as a desk or sphere, and/or the like. Once both the inputmotion and the static keys are fixed to the same object (e.g., plane,surface), by moving the finger, the user may allow the input device toswipe along the [X,Y] coordinates within such a coordinate system, assuch: the [X,Y] coordinates of the finger are returned from the firstphenomenon interface 1302 (e.g., via the depth map 1308) and are updatedin the scene (the first augmediated-reality space 1000). As the [X,Y]coordinates of the finger change, the keys (each fixed in their locationin relation to the second phenomenon interface 1306, body or object) areintersected by the fingers. Each key that is intersected may trigger anaction event that may propagate its key ID (identifier) to one or bothof a local and a back end process. The communication with a backendprocess may be, for example, via network communications. In oneimplementation, the back end process determines what word was intendedto be written, such as via pattern recognition and/or natural languageprocessing, and returns the intended word, or word suggestions. Inanother example configuration, the [Z] parameter is not fixed, and thefinger itself may intersect with the virtual keys hovering in space. Inyet another example configuration, the virtual keys may appear asthree-dimensional shapes (e.g., bumps, spheres, cubes, buttons, lights,and/or the like) on top of a surface or object, or hovering in thecoordinate system fixed to the second phenomenon interface 1306 oruser's body, and the finger may finger press on these bumps to achievetyping. In other implementations, appendages other than or in additionto the finger (e.g., a stylus or wand, arms, hands, groups of fingers,legs, feet, and/or the like) may be employed for typing and/or otherinteraction with a virtual keyboard.

An example of a software application may include a news readingapplication (news carousel). In one implementation, the software may beconfigured to: (A) display (to the user) graphics including transparentshapes (e.g., bubbles) with article/media titles (e.g., as opaquethree-dimensional mesh rendered letters) therein; and (B) rotate thegraphics about a visually displayed axis and/or an invisible verticalaxis. The axis may be fixed relative to the second phenomenon interface1306, to the user's body, and/or to a vertical axis within a trackedroom in the first augmediated-reality space 1000. A carousel may betracked as well. In an implementation of the last case (where thevertical axis is within a tracked environment), if the user turns awayfrom the carousel, the transparent shapes encapsulating the text woulddisappear, and when the user turns back, the text would reappear in thelast location that the text occupied while in the field of view of theuser. As the text is swiped left and right, e.g., by using aninteraction technique such as a “Bunched Swipe” gesture, the articlesrevolve about the axis. The article (media) closest to the user may beselected, e.g., by using an interaction technique such as a “Squeeze toMove and Pop” gesture. Once the article is popped, the body of thearticle grows within the field of view; the article gets bigger than asingle transparent shape (e.g., bubble), or the article remains the sizeof the transparent shape; the user may read the article.

The software is configured to facilitate a user-interaction technique,such as recognition of a bunched swipe gesture. A number of fingers maybe bunched together, and this gesture may be tracked, such as bydetecting the tip of the index finger and the tip of the middle finger,and then tracking the tips together returning a set of [X,Y,Z]coordinates from the depth map 1308; a swiping gesture by the user inany direction may then, in one implementation, cause rotation, e.g., ofthe virtual carousel, about the vertical axis in that respectiverotation direction.

The software is configured to facilitate a user-interaction technique,such as a squeeze to move and pop gesture. An article within thetransparent encapsulation shape (e.g., a bubble) may be selected, andmoved based on the degree of squeezing which may, in one implementation,be determined in accordance with the following example:

Operation (1) includes recognition of a moving squeeze gesture; forexample, the tracked distance between the index finger and thumb ismeasured. If the bubble is above a certain threshold (e.g., 5centimeters), and below another threshold (e.g., 10 centimeters) theshape is moved along the direction of motion of the hand (similar todragging a mouse in three-dimensional space).

Operation (2) includes recognition of a popping squeeze gesture. If thedistance is below both the thresholds, the shape is popped, and in thiscase the body of the article grows to fill the field of view. In anexample configuration, the user sees the un-popping of a shape (i.e. theshape re-encapsulates the body of the article or media) when the bunchedswipe gesture (motion) is applied to an article or media item.

Operation (3) includes recognition of a no motion. The software isconfigured to detect if the distance is above both thresholds and theshape was in motion, and whether the shape becomes still. If the shapewas originally still, the shape will remain still.

An example of a software application may include multiple stereoscopicthree-dimensional presentations on surfaces. In one implementation, thesoftware is configured to track the user in accordance with thefollowing example: using a surface-tracking technology such as afiducial marker, or the hardware system's native physical componentdepth tracker, the position and orientation of one or more surfaces(e.g., walls) in an environment may be tracked. In one implementation,the software may be configured to present graphical images to the user,such as by the following example: a stereoscopic or otherwisetwo-dimensional and/or three-dimensional image(s) (e.g., movie, internetweb page, virtual bookshelf, virtual art gallery, and/or other mediapresentation) is fixed in relation to the one or more tracked surfacesfor viewing. In one implementation, complimentary segments of the samemovie frame or media item may be displayed on different walls and, in afurther implementation, their cumulative effect provides the illusion ofdepth in all directions for a single frame. In another implementation,all presentation types may be interacted with (e.g., to perform theoperations start, stop, forward, reverse, pause, scroll, use keyboardin, pan, zoom, rotate) using the tracked hand.

An example of a software application may include virtual sculpting. Inone implementation, this software may be configured to present a virtualshape (e.g., polyhedron such as a cube) that may be fixed relative to atracked surface and/or object (e.g., have one face coplanar with atracked physical component such as a desk), or in space in front of theuser. In one implementation, the tracked hand's skeleton may be used tosculpt the virtual object and change its configuration, by sculpting anypoint of interaction between the tracked hand and the virtual object(e.g., using a realistic clay sculpting metaphor whereby the polygon isdynamically dented inwards toward the center of mass as a fingerintersects with it). In one implementation, using a virtual menu(palette) that appears to hover on top of the user's depth-tracked arm,the user may dip his other hand into the palette that hovers above thehand and use the other hand to apply effects, such as texture, color,transparency or opacity, brightness, and/or the like to the cube. Inanother implementation, the object may be a composite of multiple otherobjects that may be grouped in layers. In such an implementation, aselection menu item may permit the tracked hand to indicate alayer/object selection, and then a sculpt/texture menu item may permitthe tracked hand to sculpt or texture the selected item.

An example of a software application includes an augmediated realityinstructor in the room with or without multiple students (users). In oneimplementation, the software may be configured to track user movements,for example: the physical component that is tracked in this case is theroom (e.g., a surface such as a floor). In an example mode of operation,the software facilitates a game of “Boxing against Muhammad Ali”, andthe user's clenched fists are tracked. In another example mode ofoperation, the software facilitates “Group yoga instruction”, and thebody of the first user is tracked by the second user's depth camera. Inone implementation, the animation of the augmediated reality instructorappears in the same position orientation and scale to both users withinworld coordinates, as their position, rotation and scale are mutuallytracked using a social tracker method.

In one implementation, the software is configured to facilitate a game“Boxing against Muhammad Ali”, in which the software is configured topresent a graphical image of a life-sized, three-dimensional (stereo orotherwise) augmediated reality instructor (such as, a virtual MohammadAli) that appears facing toward the user from a near distance as seenthrough the second phenomenon interface 1306 of the apparatus 1300. Thevirtual figure is connected and/or registered to the physical componenttracking (e.g., the floor). The virtual figure may punch the user, whomay duck and avoid a punch (thus gaining points) or may receive a punch(thus losing points), or may punch the virtual figure (thus receivingpoints) as the game progresses.

In one implementation, the software is configured to facilitate a game“Yoga Instruction”, in which the software is configured to operate afirst mode, including individual yoga instruction; on top of the floor,an augmediated reality instructor is superimposed in the augmediatedreality environment. In the case of yoga, for example, the augmediatedreality instructor is positioned at a distance from the user, facing theuser and showing the user the correct yoga motions. The user imitatesthe motions provided by the augmediated reality instructor in such anevent.

In one implementation, the software is also configured to operate asecond mode, including yoga instruction and correction; to achieve thisinteraction, one example configuration includes two or more users facingeach other at a distance above a particular threshold (such as, meters).A sensor (e.g., depth camera) of a first user operates to track askeleton of a second user, and vice versa. Next, the augmediated realityinstructor will be superimposed between the users, and based on theskeletal tracking, turn to the relevant user and correct his form andprovide relevant instruction for various yoga movements. In anotherexample configuration, another separate sensor (e.g., depth camera) mayface a single user wearing the second phenomenon interface 1306(depicted in FIG. 13), and the augmediated reality instructor maycorrect the motions of the single user based on the user's skeletalposition.

An example of a software application may include an augmediated realitystrategy, life-simulation game, simulator and/or the like that istracked to a physical surface, such as using marker-less tracking andseen in three-dimensional space. In one implementation, the software isconfigured to include a mode for a user interaction technique andtracking; the hand of the user may interact with (e.g., smack,high-five, caress, and/or the like) an avatar/structure within thevirtual world. The virtual sandbox of the game/simulator may be fixedrelative to a surface or physical component that is tracked (such as abed, a floor and/or a ceiling). The software may be further configured,in one implementation, to present a graphical image, such as, uponinteraction with the user's hand, an avatar may react with a particularanimation sequence (e.g., when “smacked”, the avatar will jump). In oneimplementation, registered user gestures are correlated to displayedoutputs, such as animation sequences, via records stored in a database(e.g., including linked gesture and output tables).

An example of a software application may include a social tracker. Inone implementation, the software is configured to track user gesturesassociated with social interactions, and may be implemented using amarker-based and/or marker-less tracking strategy (e.g., in oneimplementation of a marker, a glow-in-the-dark, or otherwise visibleunder lighting fiducial, or other physical tracker, such as may beattached to a part of the apparatus 1300). In one implementation, themarker-based tracking system is configured to improve tracking.Additionally, this approach may use a unique identifier (ID) for eachpair of glasses (e.g., in the example, each marker may be and/or have aunique ID), that may, in one implementation, be connected by the users'data via their social network username if selected and/or desired. Assuch, social network information may be exchanged between users of thissystem. Since the tracker is previously identified by the system, oneuser may know the position and orientation of another user by observinghis head (wearing glasses). This may permit, for example, a bubblegraphic popping out and becoming fixed to a user's instance of thesecond phenomenon interface 1306, with information about the user.Additionally, this ID tracker may, in one implementation, permit theidentification of users whose faces are obscured by the glasses, who mayotherwise not be able to be identified with face-identificationsoftware.

For example, in one implementation, the software (social tracker) may beconfigured to operate in a first mode (a) and a second mode (b). In thefirst mode (a), the software is configured to facilitate connectivitybetween instances of the apparatus 1300 (e.g., two way connectivityand/or multi way connectivity). In this mode, provided that the glassesare connected to a network, social network information is exchangedbetween the users via a social interface. If they are friends on asocial network, they may view common friends, photographs, etc. on theuser interface 800 in their instance of the apparatus 1300. If they arenot friends, they may request each other's friendship using guidelinessimilar to the following examples, such as may be enforced by program907.

A first guideline (for the social tracker software) includes anon-friend mode: in one implementation, the only permissible thing to dowith a non-friend (requested party) is to send a special form of friendrequest. However, before such a request is sent, information about therequested party may be shown to the requestor's instance of the secondphenomenon interface 1306. This information may arrive at therequestor's glasses during or after the requested party was tracked bythe requestor's camera. In one implementation, the information does notinclude the requested party's name, only his face, for example so thatthe same degree of information behind a non-digital face to faceexchange may occur in this system, prior to establishing friendship.Additionally, the face-only request may, in one implementation, preventand/or limit future search, stalking, inbox spamming, and/or the like.

A second guideline (for the social tracker software) includes a friendmode: in one implementation, any information that is currentlypermissible by a given social network API for exchange between friendsmay similarly be exchanged in this digital social system. The secondmode (of the social tracker software) includes interaction between auser of the apparatus 1300 and a person who does not have or use aninstance of the apparatus 1300 (one way connectivity). This may refer,in some implementations, to any situation in which party A (observerhaving the apparatus 1300) receives information, e.g., via a socialnetwork from observing the face of Party B, who is not wearing aninstance of the apparatus 1300.

Regarding privacy (for the social tracker software), the software isconfigured, in one implementation, to facilitate an accept mode, inwhich in any instance of the apparatus 1300, the user may give userpermission to be tracked bare-face (this may be toggled and disabled),in exchange for the right to track others. In one implementation, a usercannot have only one privacy privilege (i.e. either the right to track,or the right to be tracked) at any given moment, but may have both.

An example of a software application may include a racing game or aspaceship game. In one implementation, the software may be configured tofacilitate user interactions and present graphical images to the users(via the apparatus 1300). The hands are tracked and a virtual vehicledisplayed on the apparatus 1300, and the software reacts to the relativepositions of the hands in space (e.g., the vertical/horizontal offsetbetween the hands, which cumulatively act, for example, as a steeringwheel, joystick, throttle, gear shift, and/or the like). In one exampleimplementation, this may be achieved by measuring such offsets, whichare used to control the rotation of the steering wheel. In anotherexample implementation, the distance of the hands from the camera maycontrol another aspect of the vehicle's trajectory (e.g., the altitudeor pitch of an aeronautical vehicle). The fingers (e.g., thumbs) mayalso be tracked, in order to provide additional instructions for thelogic of the game (e.g., the pressing down of an upright thumb towardits first could make the spaceship shoot). In another implementation,the right thumb controls acceleration/changing gears, and the left thumbcontrols braking/changing gears.

An example of a software application may include a point cloudimplementation. In one implementation, the software may be configured totrack physical components and/or virtual components, such as in thefirst augmediated-reality space 1000 of FIG. 1A. In one implementation,component tracking includes the obtaining of a point cloud (e.g., fromthe depth sensor). The depth image, described in this example as animage which assigns a value at every pixel containing the z-depth of thecoincident scene point from a ray through that pixel, may be supplied inone implementation using a camera which is pre-calibrated with a pinholecamera model. This model supplies a focal length and principal point,defining how the sensor's lens scales pixels in real-valued metrics(e.g., millimeters). Every pixel in the depth image in this exampleproduces a three-dimensional point by subtracting the principal point,dividing by the focal length, and then scaling by the z-depth value atthat pixel. This represents that an infinite ray begins at thattwo-dimensional pixel on the image sensor plane, and is projected intothe scene, intersecting some three-dimensional physical location at adistance consistent with the z-depth value at that pixel of the depthimage. This yields a three-dimensional point cloud, per-frame, from eachimage in the two-dimensional depth sensor. The combination of pointclouds from several frames occurs in this example, as the user movesabout the environment, yielding a larger, registered three-dimensionalscene point cloud that is larger than a single image frame may collect.This may be achieved, in one implementation, using computer visiontechniques to match similar points from different frames, representingthe same physical points in the scene. By analyzing how these staticthree-dimensional point locations move in the two-dimensional image orthe three-dimensional point cloud as the user moves about theenvironment, the system deduces the motion of the user (in the form of athree-dimensional rotation matrix and three-dimensional translationvector) as well as the structure in the scene, a technique referred toin some implementations as simultaneous localization and mapping (SLAM),or structure-from-motion (SFM). In one implementation, the user may dothis in real-time, so that as the user moves about the environment,rotates his head in-place, and/or the like, the user may combinethree-dimensional point clouds from different views into a consistentcoordinate system, yielding a larger three-dimensional scene map in acommon coordinate frame over which common graphics may be placed and,e.g., held motion-less, regardless of user motion.

User Interface

The following provides a description of the user interface 800 of FIG.8A in one embodiment. The user interface 800 is configured to becontrollable by use of the apparatus 1300, such as via user gestures,motions, and/or the like. In one implementation, there are variouslevels to the user interface 800. For example, there are four levels,including: Level (1): home screen level; Level (2): folder level; Level(3): application level and file level; and, Level (4): settings level.

Level (1): Home Screen Level

An icon may represent a software application and/or data. In oneimplementation, the icon may be represented by a bubble icon or anyshaped icon. In one implementation, a bubble icon is a semi-translucentsphere resembling a bubble. The icon may include an application logo.For example, inside the bubble icon there may be a three-dimensionalopaque or mostly opaque application logo. In one example design, theratio of logo to surrounding sphere may be approximately 1:5. Together,the bubble icon and the application logo may include effects that elicitand/or are associated with bubble-like behavior such as: reflecting(e.g., at least in part) an image of the user, reflecting the sunlight,simulating air flowing on the bubble and changing bubble shape slightly,simulating another physical object pressing against the bubble andchanging its shape slightly, and/or the like.

The following is a description of some user-interaction techniquesusable with the apparatus 1300 in some embodiments.

A first user-interaction technique includes the translation and rotationtechnique, such as moving of and/or through icons displayed on the userinterface 800.

A translation and rotation method includes, in one implementation, a“Relative Hand Motion Hover-Over Pre-Selection” mode. This mode allowsnavigation quickly between the icons in the user interface 800 without aneed for depth perception of hand relative to icon. Instead, therelative direction of tracked hand motion moves a virtual indicator of aselectable icon. In an effort to provide visual feedback, in one exampledesign, the currently hovered-over (homing targeted) bubble may beslightly opaquer than its neighbors, and its hue may be more saturated.Inner illumination may be a third example design for the virtualindication of selectability. The hovered-over icon may be selected usingany of a variety of selection techniques. The hovered-over icon maychange if the hand moves in the direction of a new icon, which would inturn, become the new homing target (e.g., to select the rightmost iconfrom the particular row of icons in the field of view (FIG. 9A and FIG.10A), the user moves the tracked hand right). To move to a new row oficons (e.g., open the contents of a folder) right after the folder hasbeen selected—as the next level of icons is emerging into the field ofview—the user moves his hand away from the camera. Once the user doesso, an icon from the same column, in the next row, becomes highlighted.

Another example of the translation and rotation method may include a“Squeeze to Move Translation” mode. The tracked distance between twofingers (e.g., index finger and thumb) is measured. If this conditionand/or measurement is above a certain threshold (e.g., 5 centimeters),and below another threshold (e.g., eight centimeters) the nearest shapemay be moved, according to the direction of the tracked hand. In oneexample implementation, the edges of the icon stick to the locations ofthe two fingers, and once the fingers are spread, physically realisticwarping (of the icon) during finger motion may be effectuated. Thiscontinues to occur until the upper threshold distance is passed, afterwhich the icon remains put. In another example, the centroid between thetwo fingers is tracked, and the icons move along its path.

Another selection technique includes the popping of icons. A method forpopping of icons may include a “Squeeze to Pop Selection” gesture. Inthis example selection interaction technique, if the distance betweenfingers is below the lower threshold (e.g., 5 centimeters), thistriggers the popping of the icon, releasing of icon contents, theopening of the application whose logo is inside the icon (e.g., bubble),and/or the like. Another method for popping an icon includes a “Fingerto Pop Selection” gesture; intersection of a tracked fingertip with theicon triggers its opening/popping. Another method for popping an iconincludes a “Grab to Pop Selection” gesture; the clenching of a first maybe tracked, e.g., from a starting mode of at least two fingers (“open”),to an ending mode of all fingers closed, a closed fist, and/or the like(“closed”), e.g., to trigger the opening/popping of an icon.

In one implementation, the software of the apparatus 1300 is configuredto provide an animation technique. An example of an animation techniqueincludes a “Cycle Bubble Animation” technique. Once the apparatus 1300is turned on, a number of bubbles enter a limit cycle into the field ofview from the lower right screen, such as via an arc trajectory, to thecenter screen. Once the bubbles do so, the bubbles begin to line up in arow along the middle row of field of view in a 3×3 grid.

Another example of an animation technique includes an Inward/RolodexCycle technique, which may include the emerging of a new level of icons,once a folder icon is selected. Additionally, in the case of the homescreen level of icons (top level), for example, after the openinganimation sequence (e.g., cycle bubble animation) completes, the iconsare momentarily underneath the field of view (FIG. 8A). In this initialstate case, the Inward/Rolodex Cycle technique may commenceautomatically after a predetermined short period of time (e.g., 0.5seconds), without the selection of an icon. In one exampleimplementation, the icons of the folder (one level below the foldericon) enter the field of view by cycling rapidly [270°] about the lowerhorizontal line in the 3×3 grid from behind the line and then above ittoward the user, stopping within the lower third of the field of view.This cycle improves the speed of high frequency multiple icon selection,and provides a framework for muscle memory on the path to particular endapplications (e.g., facilitates efficient usability for endapplications). Other example implementations include, but are notlimited to, a vertical rise of the next row of icons.

The following describes an example of a method of progression at theHOME SCREEN LEVEL, in one implementation, using the followingoperations:

Operation (a) includes presenting a greeting animation (for a Cycle ofBubbles).

Operation (b) includes executing an interaction technique. If an openhand is detected by the gesture-recognition camera for at least aparticular threshold (e.g., one second) four bubble icons appear frombelow and enter the bottom third of the field of view (the bottom threesquares in the 3×3 grid (see FIG. 8A).

Operation (c) includes if the home screen icons are in the field ofview, and available, e.g., for one or more of Finger to Pop, Grab toPop, or Squeeze to Pop selection techniques (see FIG. 8C).

Level (2): Folder Level

In one implementation, the x axis lies along the bottom of the field ofview 802 (e.g., the lower line in the grid). The following is an exampleprogression at the folder level, which uses a method having thefollowing operations in one implementation:

Operation (a) includes interaction with an interaction technique. Aselection technique occurs (e.g., the “play” folder is selected).

Operation (b) includes displaying four folder icons that enter the lowerfield of view via an inward spiral (see FIG. 9A and FIG. 9B).

Operation (c) includes interacting with a “Relative Hand MotionHover-Over Pre-selection mode” gesture, so that a different icon may beselected.

Operation (d) includes determining whether another subfolder isselected. If detected, then operation is returned to operation (a);otherwise if a game icon is selected, then the operation proceeds tolevel 3 (to play the game).

Level (3): Application And File Level

An end application is a software application configured to provide(display) a leaf of the tree of icons. For example, in oneimplementation, the tree of icons may be a n-ary tree, e.g., in whicheach node has at most n children. In another implementation, the nodesmay have different maximum number of children, an unconstrained numberof children, and/or the like.

In one implementation, a spiraling logo occurs when a bubble is popped,such as via a selection technique. In one design, a three-dimensionalicon of the application begins spiraling very rapidly about a vectorfrom the lower left to the upper right as the icon moves from the lowerleft to the upper right toward the top right cell of the 3×3 grid and/orarray. During the transition, in one implementation, the icon isgradually minimized to approximately a 1:10 icon-to-cell ratio (see FIG.10B). This provides visual feedback about the state of the apparatus1300, in case a non-invasive application decides to remain open withouthaving any items in the field of view (e.g., waiting for a person tocome into the field of view), or in standby mode.

A software application is configured in one implementation to provide a“Jumping out of the Bubble” effect. When a component (displayed on theuser interface 800) associated with an application has been selected,the component may appear to be entering the field of view by scaling upfrom previously invisible miniature versions within the icon. Thefollowing is an example of progression at the application and filelevel, using the following method having the following operations in oneimplementation: Operation (a) includes spiraling logo and simultaneousopening of the application by the application jumping out of therecently popped bubble (e.g., see FIG. 10A and FIG. 10B). Operation (b)includes permitting the user interface 800 to take over the field ofview.

Level (4): Settings Level

A “Settings Selection Technique” gesture includes the following, in oneimplementation: when inside an application, contextual settings willexist in every application that wants to make use of them. One exampleof a selection method and location for settings icon is as follows: theicon may be selected by entering the hand into the field of view, in theupper half of the rightmost column of the 3×3 grid for a predeterminedamount of time (e.g., two seconds). Then a bubble will appear in the topright cell, which may later be selected, such as via Finger Selectionmethods (e.g., see FIG. 11A).

A settings screen (of the user interface 800) is a screen in which oncethe settings icon is selected; a number of vertically stacked icons (forexample, initially four) enter the rightmost column of the field ofview, such as via vertical rolodex cycling (e.g., see FIG. 11B). Theicons may, in one implementation, represent the settings iconcategories, and may be shaded with a silver hue that issemi-transparent, in one implementation.

A “Chewing Gum Toggling of Settings” (see, e.g., FIG. 11C) is configuredfor a setting selection that may include moving along a spectrum ofvalues and/or selecting a value and/or degree within a continuum (e.g.,a brightness level). In one implementation, when a setting category isselected, the icon pops and a line is formed between the location of thepopped icon and the edge of the user's finger. In one particular design,the line may be curved, and as the finger moves away from the originalicon location the line becomes straighter, as would a piece of chewinggum. In one implementation, once the line is stretched away from thestatic center of the icon, a three-dimensional number on the center ofthe magnitude line and/or vector indicates what value and/or percentageis reached (e.g., the longer the line, the higher the number). In oneimplementation, for discrete, non-continuous, non-spectral, and/or thelike settlings (e.g., discrete selection such as Wi-Fi network selectionin the Wi-Fi settings), vertical Rolodex Cycling about the right edge ofthe field of view will present additional options that may be selectedvia other gestures.

In one implementation, a meditation mode is one in which non-emergencycalls, messages, and application uses are blocked for the amount of timepre-specified, or until the button is toggled. A user interfacecomponent indicating time feedback (e.g., a sand timer) may be displayedto indicate time passed thus far.

The following is an example of progression at the settings level, in oneimplementation, and includes a method having operations similar to thefollowing example: Operation (a) includes having local settings in eachindividual application, which are accessible via a settings selectiontechnique. Operation (b) includes: displaying, once a local settings topright icon is selected, a rolodex cycle of vertically stacked settingsicons, which may cycle clockwise or counterclockwise and/or enter aright column of the 3×3 grid about the axis formed by the rightmostcolumn, similarly to the horizontal selection of a folder icon.Operation (c) includes displaying (if a spectral setting is selected) aChewing Gum Toggling of Settings. If a discrete settings item isselected, another set of icons enters the field of view for selection.

Level (5): Overarching User Interface Themes and Gestures

LEVEL 5 provides implementations for themes and gestures in variousimplementations. For example. additional interaction techniques mayinclude a gun hand motion used in a shooting game, to indicate shootinga picture or a video. Another example includes parallel two-handedraising and lowering of a flat desktop of icons, in which the iconsenter from under the field of view upwards, and back down again toremove from the field of view. An option includes rotating of a virtualknob, to change continuous settings (e.g., brightness). Another optionincludes using a zoom feature, in which motion from a flat hand whosepalm is facing up (e.g., the palm spends a predetermined amount of timeflat, before the Come to Zoom motion is initiated) to fingerssimultaneously bent (similar to the “come here” natural gesture) forzoom. Another implementation uses only one finger that changes fromstraight to bent, while the remaining finger or fingers are staticallybent. Another implementation includes the ability to hold the zoom levelconstant by removing the hand from the field of view. Another optionincludes a Zoom Out gesture. This gesture may be identical to the Cometo Zoom gesture, except that the gesture starts with all the fingersbent (e.g., clenched into a first) for a predetermined amount of time(e.g., one second). The zoom out occurs when the fingers unbend. Anotheroption includes the thumb-up gesture, which may be used for liking(indicating approval) of a picture on a social network, or within thenative operating system.

Another option for LEVEL 5 includes an elevation gesture and the iconsearch mode (e.g., see FIG. 12). Example (i) includes an elevationgesture; this is, in one implementation, an aid to the search modedetailed below, and consists of a gesture of pushing the bubbles upwardsinto the field of view with one or two hands. The pushing may be done,for example. by extending the user's forearm or forearms under the fieldof view with his palm or palms open and aimed upwards. As the user'spalm or palms move upwards into the field of view, the icons rise (e.g.,in a manner resembling the behavior of naturally occurring bubbles thatare blown upwards from below). Example (ii) includes a search mode or abird's eye view of the icons tree; this facilitates a visual search. Thefollowing provides a description of an example of a search progressionin one implementation: Operation (a) includes a visual search having anability to elevate the entire set of user-interface icons, and theirsub-folder icons (e.g., see FIG. 12) into the field of view using theelevation gesture, and dynamically scale the display such that all iconsfrom the root icons to the leaf icons are inside the field of view.Then, the lower leaves of the tree may be accessed in this fashion.Operation (b) includes manipulating the tree (e.g., where the tree isflexible to rotation and/or scaling), such as though the logos, and theicons are facing the user.

Another LEVEL 5 option includes in one implementation, having bubblesassume natural physical characteristics. For example, Operation (a):when the tracked hand moves in close proximity to the bubbles, thebubbles move faster away from the hand. However, if the bubbles arefarther away from the hand, they move progressively slower away from it.Operation (b): when a bubble icon is popped or opened using one of theselection techniques, there is a distinct “pop” sound. Operation (c): invisual-search mode, as normal icon bubble rows are blown away with saidrapid gestures, they momentarily and physically accurately blow out ofthe field of view, and in a non-natural but mirroring fashion, theyre-enter the field of view in pre-specified locations. Notifications andalerts: a variety of example techniques may be employed to gain theuser's attention in case of notification, within variousimplementations, for example:

Operation (a) includes once an application at a leaf of the filestructure tree has a notification, its icon propagates to the level ofthe tree (e.g., 4-ary tree) of icons that is currently in view, squeezesto the center of the four icons with a red hue and a number appears ontop of the traditional logo in the same scale, indicating the number ofunread messages.

Operation (b) includes propagating a red hue for a leaf's icon, uptoward whatever parent node is currently in the field of view,simultaneously inserting a miniature red bubble inside the parent'sbubble with the leaf's logo and alert number.

Operation (c) includes the selection path to the leaf application thattriggered the notification, may be either the traditional Selection pathor a shortcut selection path that is instantiated with a shortcutgesture.

Operation (d) includes a miniature bubble representing the notificationthat is triggered within a leaf application may “stick out” of theparent bubble (which is currently in the field of view), allowing for ashortcut to the leaf.

Level (6): Application Tiling on Surfaces

In one implementation, a virtual canvas is an oriented planar surface ontop of which three-dimensional virtual graphics may be overlaid. Thecanvas may be configured to add a third dimension as the canvas may beoriented, and the canvas may be seen from different angles, as apainting on a wall would look different to the human eye from differentangles and orientations. In another implementation, three-dimensionalgraphics may be placed in relation to it.

In an example scenario, a user walks into a room, for example, a livingroom, and sits on a couch (wearing the sensing and display apparatus,e.g., configured as glasses). In front of the user there is a coffeetable, a TV screen, and walls (e.g., one in front and one to the side).In one implementation, the software is configured to detect candidatecanvases which may serve as surfaces on which to overlay graphics fromthe computer. In this example case, the canvases are the coffee table,TV screen, and/or the two walls. These surfaces may be indicated, to theuser, in the second phenomenon interface 1306 with a visual indication(e.g., as semi-transparent filled oriented rectangles, indicating thesize and orientation of these canvases). In one example, all openapplications are at the bottom of the screen, shown as thumbnails/bubbleicons. Consider, for example, an email window, a Facebook window, or aMicrosoft Word document. The user may drag each application thumbnailusing one or more of the interaction techniques discussed above (e.g.,“Squeeze to Move” gesture to translate, then “Squeeze to Pop” gesturefor opening and placement), one-by-one, to each of the candidatecanvases.

In an example implementation, the software is configured to detect thechosen canvas via programmed instructions associated with a methodsimilar to the following example.

Operation (a): in one example, the user selects a thumbnail for theapplication icon that the user wishes to display, and drags thethumbnail towards appropriate given canvas. In one implementation as thethumbnail is moved along this trajectory, a line is being traced outfrom an origin (e.g., from the second phenomenon interface 1306 oruser's shoulder or base of the user's finger). This trajectory isintersected with each of the candidate planes, and a determination ismade whether the trajectory has been intersected successfully, as thethumbnail is translated toward the canvas (e.g., the translationoccurring above a predetermined distance from the icon's initiallocation toward the canvas, such as 20 centimeters). When said icon isclose enough, the software may, in one implementation, deduce whichapplication goes with which canvas, and automatically updates thegraphics onto that canvas at the correct orientation. In anotherimplementation, a user may direct which canvas to direct particulargraphics, such as by exhibiting a gesture such as a flick or a throw,directing selected graphics and/or other display elements at a givencanvas, surface, and/or the like.

Operation (b): in one example, once all applications have been assignedto a canvas, the software uses the depth map 1308 of FIG. 13 (which maybe taken as static/unchanging for this example scenario) and is able totrack the user's head in three-dimensional space (e.g., rotation andtranslation) as the user's head moves. Using the depth map 1308 as thecoordinate system, the user may then get up and physically move aboutthe environment, however the state of each canvas is maintained. Then,as the user moves between the canvases, the applications remain as theywere originally chosen (assigned), and they are updated based on theuser's current head orientation and position. In another exampleconfiguration in implementations this head-tracking/scene mappingscenario, the coordinate system of the second phenomenon interface 1306(FIG. 13) is fixed and the room/physical components' three-dimensionalmap is tracked.

Operation (b): in another example, the user may put multipleapplications on the same canvas, side-by-side. To select one of thesechosen canvases for updating, the user may, for example, use afive-finger gesture to “select” a canvas. This may specify, in oneimplementation, that the user wants to adjust this display and/or thecontents of the canvas, and the canvas may indicate graphically that itis selected for manipulation (e.g., the canvas starts blinking). Then,using a virtual vector calculated between the user's two hands, theapplication real-estate may be stretched, shrunk, or rotated (in-plane),using the magnitude, or angle of the vector.

Variations or modifications to the design and construction of theapparatus 1300, within the scope of the apparatus 1300, are possiblebased on the disclosure herein. Such variations or modifications, ifwithin the spirit of the apparatus 1300, are intended to be encompassedwithin the scope of claimed subject matter.

ADDITIONAL DESCRIPTION

The following clauses are offered as further description of the examplesof the apparatus. Any one or more of the following clauses may becombinable with any another one or more of the following clauses and/orwith any subsection or a portion or portions of any other clause and/orcombination and permutation of clauses. Any one of the following clausesmay stand on its own merit without having to be combined with anotherother of the clauses or with any portion of any other clause, etc.Clause (1): a sensing and display apparatus, comprising: an interfaceassembly including: a first interface module configured to exchangesensor signals and effector signals with a first space; and a secondinterface module configured to exchange effector signals with a secondspace, the effector signals being user presentable, at least in part, inat least one of the first space and the second space. Clause (2): asensing and display apparatus, comprising: an interface assemblyincluding: a first interface module configured to interface with atleast one sensor signal representing sensory phenomenon received from afirst space; and a second interface module configured to interface withat least one effector signal representing sensory phenomenon to thefirst space and to the second space. Clause (3): the apparatus of anyclause mentioned in this paragraph, wherein: the at least one sensorsignal and the at least one effector signal are presentable, at least inpart, for user-sensory consumption in any one of the first space and thesecond space. Clause (4): the apparatus of any clause mentioned in thisparagraph, further comprising: a processing apparatus being operativelycoupled to the interface assembly, and configured to process the atleast one sensor signal being interfaced with the interface assembly,and to process the at least one effector signal being interfaced withthe interface assembly. Clause (5): the apparatus of any clausementioned in this paragraph, further comprising: sensory-phenomenonsensors configured to transmit the at least one sensor signal derivedfrom sensory phenomena received from the first space, and alsoconfigured to transmit the at least one sensor signal derived fromsensory phenomena received from the second space. Clause (6): theapparatus of any clause mentioned in this paragraph, further comprising:sensory-phenomenon effectors configured to transmit the at least oneeffector signal associated with sensory phenomena to the first space,and also configured to transmit the at least one effector signalassociated with sensory phenomenon to the second space. Clause (7): theapparatus of any clause mentioned in this paragraph, wherein: theinterface assembly is configured to interface with the at least onesensor signal representing sensory phenomena received, via thesensory-phenomenon sensors, from the first space, and also representingsensory phenomena received from the second space, and also configured tointerface with the at least one effector signal representing sensoryphenomena, via sensory-phenomenon effectors, to the first space and tothe second space. Clause (8): the apparatus of any clause mentioned inthis paragraph, wherein: the first space is configured to be accessibleby users; and the second is configured to be accessible by a singleuser. Clause (9): the apparatus of any clause mentioned in thisparagraph, wherein: the at least one sensor signal is derived from anyone of an audio sensory phenomenon, a visual sensory phenomenon, and atactile sensory phenomenon. Clause (10): the apparatus of any clausementioned in this paragraph, wherein: the at least one effector signalis derived from any one of an audio sensory phenomenon, a visual sensoryphenomenon, and a tactile sensory phenomenon. Clause (11): the apparatusof any clause mentioned in this paragraph, wherein: thesensory-phenomenon sensors and the sensory-phenomenon effectors include:a first sensory-phenomenon sensor configured to transmit a sensor signalderived from sensory phenomena from the first space; and a firstsensory-phenomenon effector configured to transmit an effector signalassociated with sensory phenomena to the first space. Clause (12): theapparatus of any clause mentioned in this paragraph, wherein: thesensory-phenomenon sensors and the sensory-phenomenon effectors include:a second sensory-phenomenon effector configured to transmit an effectorsignal having sensory phenomena to the second space; and a secondsensory-phenomenon sensor configured to transmit a sensor signal derivedfrom the sensory phenomena from the second space. Clause (13): theapparatus of any clause mentioned in this paragraph, wherein: the firstsensory-phenomenon sensor and the first sensory-phenomenon effector facea direction of a field of view of the user in the first space; and thesecond sensory-phenomenon effector and the second sensory-phenomenonsensor face a direction toward eyes of the user in the second space.Clause (14): the apparatus of any clause mentioned in this paragraph,wherein: the processing apparatus is configured to route the sensorsignals between the first space and the second space. Clause (15): theapparatus of any clause mentioned in this paragraph, wherein: theprocessing apparatus is configured to route the effector signals betweenthe first space and the second space. Clause (16): the apparatus of anyclause mentioned in this paragraph, wherein: the interface assemblyincludes: the first interface module configured to interface with afirst sensory-phenomenon sensor, the first sensory-phenomenon sensorconfigured to transmit a sensor signal derived from sensory phenomenareceived by the first sensory-phenomenon sensor from the first space;and the second interface module configured to interface with a firstsensory-phenomenon effector, the first sensory-phenomenon effectorconfigured to transmit an effector signal associated with sensoryphenomena to the first space. Clause (17): the apparatus of any clausementioned in this paragraph, wherein: the interface assembly includes: athird interface module configured to interface with a secondsensory-phenomenon effector, the second sensory-phenomenon effectorconfigured to transmit an effector signal having sensory phenomena tothe second space; and a fourth interface module configured to interfacewith a second sensory-phenomenon sensor, the second sensory-phenomenonsensor configured to transmit a sensor signal derived from the sensoryphenomena received by the second sensory-phenomenon sensor from thesecond space. Clause (18): the apparatus of any clause mentioned in thisparagraph, wherein: the processing apparatus is configured to route asensor signal received from a first sensory-phenomenon sensor to thesecond space via a second sensory-phenomenon effector. Clause (19): theapparatus of any clause mentioned in this paragraph, wherein: theprocessing apparatus is configured to route a sensor signal receivedfrom a second sensory-phenomenon sensor to the first space via a firstsensory-phenomenon effector. Clause (20: the apparatus of any clausementioned in this paragraph, further comprising: a user-wearableinterface configured to facilitate user wearing. Clause (21): theapparatus of any clause mentioned in this paragraph, further comprising:a frame assembly being configured to fixedly support any one of:sensory-phenomenon sensors configured to transmit the at least onesensor signal derived from sensory phenomena received from the firstspace, and also configured to transmit the at least one sensor signalderived from sensory phenomena received from the second space; andsensory-phenomenon effectors configured to transmit the at least oneeffector signal associated with sensory phenomena to the first space,and also configured to transmit the at least one effector signalassociated with sensory phenomena to the second space. Clause (22): theapparatus of any clause mentioned in this paragraph, further comprising:a frame assembly being configured to fixedly support any one of: ahead-mountable assembly; a digital eye glass; a LiDAR unit; a visionsystem; an optical sensor; a display unit; a removable shade; aninfrared transmitter; an infrared receiver; and a geophone. Clause (23):a method, comprising: receiving a sensor signal representing at leastone sensory phenomenon received from a first space and from a secondspace; and providing an effector signal representing at least oneeffectory phenomenon to the first space and to the second space. Clause(24): the method of any clause mentioned in this paragraph, furthercomprising: processing the sensor signal and the effector signal. Clause(25): the method of any clause mentioned in this paragraph, furthercomprising: routing the sensor signal between the first space and thesecond space. Clause (26): the method of any clause mentioned in thisparagraph, further comprising: routing the effector signal between thefirst space and the second space. Clause (27): a display apparatus,comprising: an interface assembly configured to interface with a firstspace and with a second space, and configured to convey sensor signalsand effector signals associated with the first space and the secondspace; a processing apparatus operatively coupled to the interfaceassembly and configured to process the sensor signals and the effectorsignals conveyed by the interface assembly; and a memory assemblyconfigured to tangibly embody a processing program including a sequenceof programmed instructions configured to direct the processing apparatusto execute operations on the sensor signals and the effector signals.Clause (28): a user interface, comprising: a first interface sectionconfigured to display phenomena derived from a first space; and a secondinterface section configured to display phenomena derived from a secondspace. Clause (29: an apparatus, comprising: a first phenomenoninterface configured to operatively interface with a first space; and asecond phenomenon interface configured to operatively interface with asecond space. Clause (30: the apparatus of any clause mentioned in thisparagraph, wherein: the first phenomenon interface includes a firstsensory phenomenon effector and a first sensory phenomenon sensor eachconfigured to operatively interact with the first space. Clause (31: theapparatus of any clause mentioned in this paragraph, wherein: the secondphenomenon interface includes a second sensory phenomenon effector and asecond sensory phenomenon sensor each configured to operatively interactwith the second space. Clause (32: the apparatus of any clause mentionedin this paragraph, further comprising: a processing assembly configuredto operatively couple to the first phenomenon interface and the secondphenomenon interface, and further configured to operatively interactwith the first sensory phenomenon effector, the first sensory phenomenonsensor, the second sensory phenomenon effector, and the second sensoryphenomenon sensor. Clause (33): a manifoldizer for the apparatus of anyclause mentioned in this paragraph, said manifoldizer comprising: agesture sensor for sensing a gesture of a body part of a wearer of theapparatus; a manifold generator responsive to a manifold startinggesture detected by the gesture sensor; a manifold contact detector fordetection of contact between the wearer and a manifold; a manifolddisplay generator responsive to an output of the manifold contactgenerator. Clause (34): a toposculpter, including the manifoldizer ofany clause mentioned in this paragraph, and further including: amanifold integrator responsive to a substantially continuous manifoldcontinuation gesture. Clause (35): A manifoldized display generator,including the manifoldizer of any clause mentioned in this paragraph,the manifold display generator for displaying an output of a virtual orreal device. Clause (36). The manifoldized display generator of anyclause mentioned in this paragraph, wherein: said manifold displaygenerator displays a two-dimensional manifold comprising a rigid planarpatch, embedded in a three-dimensional augmediated environment visibleto said wearer of said digital eye glass. Clause (37). An abakographicuser-interface comprising: a three-dimensional vision system, aprocessor, and a display, the processor including anexposure-integrator, the exposure integrator responsive to an input fromthe three-dimensional vision system. Clause (38). The user-interface ofany clause mentioned in this paragraph, including: a manifold-dimensionsensor and a gesture sensor, the gesture sensor responsive to an outputof the manifold dimension sensor. Clause (39). The user-interface of anyclause mentioned in this paragraph, wherein: the real-world physicalobject comprises a two dimensional manifold, and wherein the manifolddimension sensor determines whether a gesture is made along anabakograph, to an abakograph from within the manifold, or to anabakograph from within three-dimensional space not on the manifold.Clause (40): A three-dimensional sculpting system for generatingmanifolds in space, including: a three-dimensional vision system; anexposure integrator; and a display for showing an output of the exposureintegrator, the exposure integrator accumulating exposures. Clause (41):The system of any clause mentioned in this paragraph, further including:a gesture sensor for sensing a gesture for initiating an exposureaccumulation of the exposure integrator. Clause (42): the system of anyclause mentioned in this paragraph, wherein: the gesture sensor forsensing a gesture for terminating an exposure accumulation of theexposure integrator. Clause (43). The system of any clause mentioned inthis paragraph, wherein: the exposure integrator for generating atemporally growing manifold in three-dimensional space. Clause (44). Thesystem of any clause mentioned in this paragraph, further comprising: atangency sensor for determining a tangency of a body part of a user to apath defined by the temporally growing manifold. Clause (45). A methodfor abakographic compositions, the method comprising: allocating andclearing an exposure buffer; sensing a degree of exposure strength of anexposer; sensing a location, in three-dimensional space, of an exposer;adding to the exposure buffer, by an amount proportional to the exposurestrength, in association with the location of the exposer. Clause (46).The method of any clause mentioned in this paragraph, wherein: thedegree of exposure strength being a binary quantity determined by agesture sensor. Clause (47). The method of any clause mentioned in thisparagraph, wherein: the exposer is a light source, and the exposurestrength is a degree of illumination of the light source. Clause (48).The method of any clause mentioned in this paragraph, wherein: theexposer is a fingertip, and the exposure strength is determined by acontinuously variable hand gesture. Clause (49). The method of anyclause mentioned in this paragraph, further comprising: initiating by agesture sensor an abakograhic composition, the gesture sensor sensing a“gun” gesture of an index finger and thumb. Clause (50). The method ofany clause mentioned in this paragraph, wherein: the exposure strengthcontinuously variable by an angle between the thumb and the indexfinger. Clause (51). A method for toposculpting, the method comprising:allocating and clearing a toposculpting buffer, the method comprising:sensing a degree of exposure strength of an exposer; sensing a location,in three-dimensional space, of an exposer; adding to the toposculptingbuffer, by an amount proportional to the exposure strength, inassociation with the location of the exposer. Clause (52). The method ofany clause mentioned in this paragraph, wherein: sensing the locationincludes sensing tangency of a planar portion of a body part of a userof the method, to a toposculpture in progress, at a point of contactwith the toposculpture at its most recent point, the exposure strengthproportional to an output of the sensing tangency. Clause (53). Aspaceglass apparatus for viewing scannable subject matter, comprising: aspatial imaging camera; a processor responsive to at least one output ofthe spatial imaging camera; a spatial imaging display selected from thegroup consisting of a holographic video display, a stereoscopic videodisplay, and an aremac, stereoscopic display, wherein the spatialimaging display is responsive to at least one output of the processor,and wherein the spatial imaging display provides a Point-of-Eye imagederived from at least some of the scannable subject matter. Clause (54).The apparatus of any clause mentioned in this paragraph, wherein thespatial imaging camera comprises a 3D camera. Clause (55): The apparatusof any clause mentioned in this paragraph, wherein: the processorincludes a taction detector. Clause (56). The apparatus of any clausementioned in this paragraph, wherein: the processor includes ahomography in-trusion detector. Clause (57). The apparatus of any clausementioned in this paragraph, wherein: the processor includes a sphericalvolumetric intrusion detector. Clause (58). The apparatus of any clausementioned in this paragraph, wherein: the processor includes a bubblemetaphor generator. Clause 59: The apparatus of any clause mentioned inthis paragraph, wherein: the processor simulates a bubble menu, and abubble menu selector responsive to an intrusion into a volume of asphere of radius r located a distance of less than r from a part of abody of a user of the apparatus, the part of the body of the user beingstatistically significant in its degree of intrusion into the volume.Clause 60: A shared augmediated reality system for spaceglasses,comprising: a first spaceglass for use by a first user of the sharedaugmediated reality system, wherein the first spaceglass comprises afirst spatial imaging camera; a second spaceglass for use by a seconduser of the shared augmediated reality system, wherein the secondspaceglass comprises a second spatial imaging camera, wherein the firstand second spatial imaging cameras are configured for spatial imaging ofa subject matter; and a spatial imaging multiplexer for multiplexing ascanning from the first and second spatial imaging cameras. Clause 61:The shared augmediated reality system of any clause mentioned in thisparagraph, wherein: the spatial imaging multiplexer is a time-divisionmultiplexer. Clause 62: A digital eye glass apparatus for being viewedby one or both eyes of a wearer of the digital eye glass, comprising: atleast one sensor for sensing light from a plurality of differentdirections; a processor responsive to at least one input from saidsensor; and a display responsive to an output of said processor, whereinsaid display provides a point-of-eye image. Clause 63: The digital eyeglass apparatus of any clause mentioned in this paragraph, wherein: saidsensor is a 3D camera. Clause 64: The digital eye glass apparatus of anyclause mentioned in this paragraph, wherein: the 3D camera implements 3DHDR imaging. Clause 65: The digital eye glass apparatus of any clausementioned in this paragraph, wherein: the 3D camera includes atime-division multiplexer in cooperation with other 3D cameras worn byother participants in a shared computer-mediated reality environment.Clause 66: The digital eye glass apparatus of any clause mentioned inthis paragraph, wherein: the 3D camera includes a variable amplitudemultiplexer in cooperation with other 3D cameras worn by otherparticipants in a shared computer-mediated reality environment, thevariable amplitude multiplexer providing approximately identicalgettings of subject matter in response to different degrees ofillumination of the subject matter. Clause 67: The digital eye glassapparatus of any clause mentioned in this paragraph, wherein: theprocessor includes a synthetic aperture imaging constructor responsiveto a lightspace generated by illuminators of more than one participantsharing a computer-mediated reality. Clause 68: A spaceglass system,comprising: one or more spaceglasses for spatially imaging subjectmatter in view of the spaceglasses, and further comprising: at least one3D camera; a processor responsive to at least one input from the atleast one 3D camera; a display responsive to an output of saidprocessor, the display providing an EyeTap point-of-view renderingresponsive to spatial information present in the subject matter. Clause69: The system of any clause mentioned in this paragraph, wherein: theprocessor includes a comparametric compositor. Clause 70: The system ofany clause mentioned in this paragraph, further comprising: a firstspaceglass and a second spaceglass, and wherein the processor is usedwith the second spaceglass. Clause 71: The system of any clausementioned in this paragraph, further comprising: a first spaceglass anda second spaceglass, and wherein the processor is used with the secondspaceglass, and wherein the first and second spaceglasses are worn bytwo separate users, and wherein the processor including asuperposimetric compositor. Clause 72: A spatial imaging devicecomprising: a lightspace analysis glass; a processor responsive to anoutput of said lightspace analysis glass; a lightspace synthesis glassresponsive to an output of said processor; and a lightspacecollinearizer. Clause 73: A gesture recognizing processor-implementedmethod, comprising: tracking a first environmental object in anenvironment via at least one sensor; assigning at least oneenvironmental object identifier to the first environmental object;identifying at least one spatial coordinate associated with the firstenvironmental obj ect; associating a coordinate system with theenvironmental obj ect identifier based on the at least one spatialcoordinate. Clause 74: The method of any clause mentioned in thisparagraph, further comprising: mapping a gesture of a secondenvironmental object in relation to the coordinate system via the atleast one sensor. Clause 75: The method of any clause mentioned in thisparagraph, wherein: mapping a gesture further comprises: generating apoint cloud in association with the second environmental object. Clause76: The method of any clause mentioned in this paragraph, wherein:mapping a gesture further comprises: identifying at least one secondenvironmental object spatial coordinate of the second environmentalobject; and determining a relation between the second environmentalobject spatial coordinate and the spatial coordinate of the firstenvironmental object. Clause 77: The method of any clause mentioned inthis paragraph, wherein: the relation comprises a relative displacement.Clause 78: The method of any clause mentioned in this paragraph,wherein: the relation comprises a relative orientation. Clause 79: Themethod of any clause mentioned in this paragraph, wherein: the sensorcomprises a depth camera. Clause 80: The method of any clause mentionedin this paragraph, wherein: identifying the at least one secondenvironmental object spatial coordinate comprises: generating a depthmap in association with the second environmental object. Clause 81: Themethod of any clause mentioned in this paragraph, further comprising:generating an extended depth map by combining the depth map associatedwith the second environmental object with at least one supplementaldepth map. Clause 82: The method of any clause mentioned in thisparagraph, wherein: determining a relation between the secondenvironmental object spatial coordinate and the spatial coordinate ofthe first environmental object further comprises: comparing the depthmap associated with the second environmental object with a firstenvironmental object depth map. Clause 83: The method of any clausementioned in this paragraph, wherein: the first environmental object isa frame. Clause 84: The method of any clause mentioned in thisparagraph, wherein: the first environmental object is a surface. Clause85: The method of any clause mentioned in this paragraph, wherein: thesurface is a table surface. Clause 86: The method of any clausementioned in this paragraph, wherein: the surface is a wall. Clause 87:The method of any clause mentioned in this paragraph, wherein: thesurface is a display screen. Clause 88: The method of any clausementioned in this paragraph, wherein: the second environmental object isa user body part. Clause 89: The method of any clause mentioned in thisparagraph, wherein the user body part is a user's palm and thumb, andwherein identifying at least one second environmental object spatialcoordinate comprises identifying at least one relative position of theuser's palm and thumb. Clause 90: The method of any clause mentioned inthis paragraph, wherein: tracking a first environmental object in anenvironment further comprises: generating at least one point cloudassociated with the first environmental object. Clause 91: The method ofany clause mentioned in this paragraph, further comprising: generatingan extended point cloud by combining the at least one point cloudassociated with the first environmental object with at least onesupplemental point cloud. Clause 92: An apparatus, comprising: at leastone input sensor; at least one output display; at least onecomputational device communicatively coupled to the at least one inputsensor and at least one output display, wherein the at least onecomputational devices is configured to issue processor-readable programinstructions, comprising: determining the spatial position of at leastone appendage via the at least one input sensor; selecting at least onedisplay item based on the spatial position; and presenting the at leastone display item via the at least one output display. Clause 93: Anapparatus, comprising: at least one sensor configured to detect therelative spatial position of at least a first environmental obj ect anda second environmental object; at least one computational device issuingprogram instructions configured to recognize the relative spatialposition and configure at least one display component; and at least oneoutput display configured to display the at least one display component.Clause 94: The apparatus of any clause mentioned in this paragraph,wherein: the at least one sensor comprises a depth camera configured toproduce a depth map. Clause 95: The apparatus of any clause mentioned inthis paragraph, wherein the computational device recognizes the relativespatial position based on the depth map. Clause 96: The apparatus of anyclause mentioned in this paragraph, wherein the computational devicerecognizes the relative spatial position based on first and second pointclouds associated, respectively, with the first environmental object andthe second environmental object. Clause 97: The apparatus of any clausementioned in this paragraph, wherein the computational device recognizesthe relative spatial position as a touch gesture based on intersectionof the first and second point clouds. Clause 98: The apparatus of anyclause mentioned in this paragraph wherein the at least one displaycomponent comprises a virtual keyboard and the second environmentalobject comprises a user body part. Clause 99: The apparatus of anyclause mentioned in this paragraph, wherein the first physical object isa surface and wherein the virtual keyboard is anchored to a coordinatesystem associated with that surface within the at least one outputdisplay. Clause 100. The apparatus of any clause mentioned in thisparagraph, wherein the relative spatial position further includes arelative spatial motion. Clause 101. The apparatus of any clausementioned in this paragraph, wherein the relative spatial motioncomprises a swipe. Clause 102. The apparatus of any clause mentioned inthis paragraph, wherein the swipe is a bunched swipe. Clause 103. Theapparatus of any clause mentioned in this paragraph, wherein therelative spatial motion comprises at least one of a moving squeeze and apopping squeeze. Clause 104. The apparatus of any clause mentioned inthis paragraph, wherein the first environmental object comprises asurface, wherein the at least one computational device is configured totrack the surface, and wherein the at least one output display isconfigured to fix at least one media presentation to the trackedsurface. Clause 105. The apparatus of any clause mentioned in thisparagraph, wherein the at least one media presentation is fixed in astereoscopic presentation. Clause 106. The apparatus of any clausementioned in this paragraph, wherein the at least one media presentationincludes application data. Clause 107. The apparatus of any clausementioned in this paragraph, wherein the relative spatial positionindicates an association between the surface and the at least one mediapresentation. Clause 108. The apparatus of any clause mentioned in thisparagraph, wherein the relative spatial position includes adrag-and-drop of the at least one media presentation to the surface.Clause 109. The apparatus of any clause mentioned in this paragraph,wherein the second environmental object comprises a gesturing hand,wherein the first environmental object comprises a virtual model in avirtual environment, and wherein the at least one computational deviceissues program instructions to configure the at least one displaycomponent representing the virtual model by sculpting any point ofcontact between the gesturing hand and the virtual model. Clause 110.The apparatus of any clause mentioned in this paragraph, wherein the atleast one computational device issues program instructions to associateat least one unique identifier to at least one of the firstenvironmental object and the second environmental object, the uniqueidentifier being connected to social network data associated with auser. Clause 111. The apparatus of any clause mentioned in thisparagraph, wherein the program instructions to associate at least oneunique identifier include instructions to apply a marker to at least oneof the first environmental object and the second environmental objectvia the at least one output display. Clause 112. The apparatus of anyclause mentioned in this paragraph, wherein the computational device isconfigured to issue program instructions to recognize the uniqueidentifier and wherein the output display is configured to display atleast some of the social network data associated with the user. Clause113. The apparatus of any clause mentioned in this paragraph, whereinthe at least one sensor is configured to detect a thumbs-up usergesture, and the at least one computational device issues programinstructions to associate a feedback with the unique ID. Clause 114. Theapparatus of any clause mentioned in this paragraph, configured as apair of glasses. Clause 115. A user interface managingprocessor-implemented method, comprising: providing a plurality of iconsfor display via at least one head-mounted output display, the pluralityof icons configured for selection via at least one gesture trackingsensor; receiving user gesture data via the at least one gesturetracking sensor, the user gesture data indicating selection of aselected icon of the plurality of icons; accessing a hierarchy of iconsbased on the selection of the selected icon; and providing part of thehierarchy of icons for display via the at least one head-mounted outputdisplay based on the selected icon. Clause 116. The method of any clausementioned in this paragraph, wherein providing the plurality of iconsfor display further occurs only when the at least one gesture trackingsensor senses an icon summoning gesture. Clause 117. The method of anyclause mentioned in this paragraph, wherein the icon summoning gesturecomprises presenting an open hand for at least a threshold period oftime. Clause 118. The method of any clause mentioned in this paragraph,wherein the plurality of icons are presented using at least one cyclebubble animation. Clause 119. The method of any clause mentioned in thisparagraph, wherein the icons are bubble icons. Clause 120. The method ofany clause mentioned in this paragraph, wherein the user gesture datacomprises a relative hand motion hover-over pre-selection. Clause 121.The method of any clause mentioned in this paragraph, wherein the usergesture data comprises a squeeze-to-move translation. Clause 122. Themethod of any clause mentioned in this paragraph, wherein the usergesture data comprises a squeeze-to-pop selection. Clause 123. Themethod of any clause mentioned in this paragraph, wherein at least oneof the plurality of icons includes an application logo. Clause 124. Themethod of any clause mentioned in this paragraph, wherein: providingpart of the hierarchy of icons for display via the at least onehead-mounted output display includes: displaying at least one animationof the selected icon in response to the user gesture data. Clause 125.The method of any clause mentioned in this paragraph, wherein: the atleast one animation comprises an inward/rolodex cycle. Clause 126. Themethod of any clause mentioned in this paragraph, wherein: the hierarchyof icons include a folder level and an application/file level. Clause127. The method of any clause mentioned in this paragraph, wherein: thehierarchy of icons includes a settings level. Clause 128. The method ofany clause mentioned in this paragraph, wherein: the hierarchy of iconsfurther includes a settings sublevel associated with at least onedisplay parameter. Clause 129. The method of any clause mentioned inthis paragraph, wherein: the at least one display parameter comprises acontinuous display parameter, and further comprising: receivingselection of the at least one settings sublevel icon associated with theat least one display parameter via the at least one gesture trackingsensor; determining a relative spatial position of the settings sublevelicon and at least one user body party; providing via the output displayan interface element comprising a line stretching from the settingssublevel icon to the user body part; and adjusting the at least onedisplay parameter and a length of the line based on the relative spatialposition. Clause 130. A gesture recognizing apparatus, comprising: amemory; a processor coupled to the memory and configured to issue aplurality of program instructions stored in the memory, the programinstructions comprising: track a first environmental obj ect in anenvironment via at least one sensor; assign at least one environmentalobj ect identifier to the first environmental object; identify at leastone spatial coordinate associated with the first environmental object;associate a coordinate system with the environmental object identifierbased on the at least one spatial coordinate. Clause 131. The apparatusof any clause mentioned in this paragraph, further comprising: map agesture of a second environmental object in relation to the coordinatesystem via the at least one sensor. Clause 132. The apparatus of anyclause mentioned in this paragraph, wherein map a gesture furthercomprises: generate a point cloud in association with the secondenvironmental object. Clause 133. The apparatus of any clause mentionedin this paragraph, wherein map a gesture further comprises: identifyingat least one second environmental object spatial coordinate of thesecond environmental obj ect; and determining a relation between thesecond environmental object spatial coordinate and the spatialcoordinate of the first environmental object. Clause 134. The apparatusof any clause mentioned in this paragraph, wherein the relationcomprises a relative displacement. Clause 135. The apparatus of anyclause mentioned in this paragraph, wherein the relation comprises arelative orientation. Clause 136. The apparatus of any clause mentionedin this paragraph, wherein the sensor comprises a depth camera. Clause137. The apparatus of any clause mentioned in this paragraph, whereinidentify the at least one second environmental object spatial coordinatecomprises: generate a depth map in association with the secondenvironmental object. Clause 138. The apparatus of any clause mentionedin this paragraph, further comprising: generate an extended depth map bycombining the depth map associated with the second environmental objectwith at least one supplemental depth map. Clause 139. The apparatus ofany clause mentioned in this paragraph, wherein; determine a relationbetween the second environmental object spatial coordinate and thespatial coordinate of the first environmental object further comprises:compare the depth map associated with the second environmental objectwith a first environmental object depth map. Clause 140. The apparatusof any clause mentioned in this paragraph, wherein the firstenvironmental object is a frame. Clause 141. The apparatus of any clausementioned in this paragraph, wherein the first environmental object is asurface. Clause 142. The apparatus of any clause mentioned in thisparagraph, wherein the surface is a table surface. Clause 143. Theapparatus of any clause mentioned in this paragraph, wherein the surfaceis a wall. Clause 144. The apparatus of any clause mentioned in thisparagraph, wherein the surface is a display screen. Clause 145. Theapparatus of any clause mentioned in this paragraph, wherein the secondenvironmental object is a user body part. Clause 146. The apparatus ofany clause mentioned in this paragraph, wherein: the user body part is auser's palm and thumb, and wherein identifying at least one secondenvironmental object spatial coordinate comprises identifying at leastone relative position of the user's palm and thumb. Clause 147. Theapparatus of any clause mentioned in this paragraph, wherein tracking afirst environmental object in an environment further comprises:generating at least one point cloud associated with the firstenvironmental object. Clause 148. The apparatus of any clause mentionedin this paragraph, further comprising: generating an extended pointcloud by combining the at least one point cloud associated with thefirst environmental object with at least one supplemental point cloud.Clause 149. A user interface managing apparatus, comprising: a memory; aprocessor coupled to the memory and configured to issue a plurality ofprogram instructions stored in the memory, the program instructionscomprising: provide a plurality of icons for display via at least onehead-mounted output display, the plurality of icons configured forselection via at least one gesture tracking sensor; receive user gesturedata via the at least one gesture tracking sensor, the user gesture dataindicating selection of a selected icon of the plurality of icons;access a hierarchy of icons based on the selection of the selected icon;and provide part of the hierarchy of icons for display via the at leastone head-mounted output display based on the selected icon. Clause 150.The apparatus of any clause mentioned in this paragraph, whereinproviding the plurality of icons for display further occurs only whenthe at least one gesture tracking sensor senses an icon summoninggesture. Clause 151. The apparatus of any clause mentioned in thisparagraph, wherein the icon summoning gesture comprises presenting anopen hand for at least a threshold period of time. Clause 152. Theapparatus of any clause mentioned in this paragraph, wherein theplurality of icons are presented using at least one cycle bubbleanimation. Clause 153. The apparatus of any clause mentioned in thisparagraph, wherein the icons are bubble icons. Clause 154.The apparatusof any clause mentioned in this paragraph, wherein the user gesture datacomprises a relative hand motion hover-over pre-selection. Clause 155.The apparatus of any clause mentioned in this paragraph, wherein theuser gesture data comprises a squeeze-to-move translation. Clause 156.The apparatus of any clause mentioned in this paragraph, wherein theuser gesture data comprises a squeeze-to-pop selection. Clause 157. Theapparatus of any clause mentioned in this paragraph, wherein at leastone of the plurality of icons includes an application logo. Clause 158.The apparatus of any clause mentioned in this paragraph, wherein:provide part of the hierarchy of icons for display via the at least onehead-mounted output display includes: display at least one animation ofthe selected icon in response to the user gesture data. Clause 159. Theapparatus of any clause mentioned in this paragraph, wherein the atleast one animation comprises an inward/rolodex cycle. Clause 160.Theapparatus of any clause mentioned in this paragraph, wherein thehierarchy of icons include a folder level and an application/file level.Clause 161. The apparatus of any clause mentioned in this paragraph,wherein the hierarchy of icons includes a settings level. Clause 162.The apparatus of any clause mentioned in this paragraph, wherein thehierarchy of icons further includes a settings sublevel associated withat least one display parameter. Clause 163. The apparatus of any clausementioned in this paragraph, wherein the at least one display parametercomprises a continuous display parameter, and further comprising:receive selection of the at least one settings sublevel icon associatedwith the at least one display parameter via the at least one gesturetracking sensor; determine a relative spatial position of the settingssublevel icon and at least one user body party; provide via the outputdisplay an interface element comprising a line stretching from thesettings sublevel icon to the user body part; and adjust the at leastone display parameter and a length of the line based on the relativespatial position. Clause 164. A gesture recognizing non-transitorymedium, comprising: program instructions issuable by a processor coupledto the medium to cause the processor to: track a first environmentalobject in an environment via at least one sensor; assign at least oneenvironmental object identifier to the first environmental object;identify at least one spatial coordinate associated with the firstenvironmental object; associate a coordinate system with theenvironmental obj ect identifier based on the at least one spatialcoordinate. Clause 165.The medium of any clause mentioned in thisparagraph, further comprising: map a gesture of a second environmentalobject in relation to the coordinate system via the at least one sensor.Clause 166. The medium of any clause mentioned in this paragraph,wherein; map a gesture further comprises: generate a point cloud inassociation with the second environmental object. Clause 167. The mediumof any clause mentioned in this paragraph, wherein map a gesture furthercomprises: identify at least one second environmental obj ect spatialcoordinate of the second environmental object; and determining arelation between the second environmental object spatial coordinate andthe spatial coordinate of the first environmental object. Clause 168.The medium of any clause mentioned in this paragraph, wherein therelation comprises a relative displacement. Clause 169. The medium ofany clause mentioned in this paragraph, wherein the relation comprises arelative orientation. Clause 170. The medium of any clause mentioned inthis paragraph, wherein the sensor comprises a depth camera. Clause 171.The medium of any clause mentioned in this paragraph, wherein: identifythe at least one second environmental object spatial coordinatecomprises: generate a depth map in association with the secondenvironmental object. Clause 172. The medium of any clause mentioned inthis paragraph, further comprising: generate an extended depth map bycombining the depth map associated with the second environmental objectwith at least one supplemental depth map. Clause 173. The medium of anyclause mentioned in this paragraph, wherein: determine a relationbetween the second environmental object spatial coordinate and thespatial coordinate of the first environmental object further comprises:compare the depth map associated with the second environmental objectwith a first environmental object depth map. Clause 174. The medium ofany clause mentioned in this paragraph, wherein the first environmentalobject is a frame. Clause 175. The medium of any clause mentioned inthis paragraph, wherein the first environmental object is a surface.Clause 176. The medium of any clause mentioned in this paragraph,wherein the surface is a table surface. Clause 177. The medium of anyclause mentioned in this paragraph, wherein the surface is a wall.Clause 178.The medium of any clause mentioned in this paragraph. whereinthe surface is a display screen. Clause 179. The medium of any clausementioned in this paragraph, wherein the second environmental object isa user body part. Clause 180.The medium of any clause mentioned in thisparagraph, wherein the user body part is a user's palm and thumb, andwherein identifying at least one second environmental object spatialcoordinate comprises identifying at least one relative position of theuser's palm and thumb. Clause 181. The medium of any clause mentioned inthis paragraph, wherein tracking a first environmental object in anenvironment further comprises: generating at least one point cloudassociated with the first environmental object. Clause 182.The medium ofany clause mentioned in this paragraph, further comprising: generatingan extended point cloud by combining the at least one point cloudassociated with the first environmental object with at least onesupplemental point cloud. Clause 183. A user interface managing medium,comprising: program instructions issuable by a processor coupled to themedium to cause the processor to: provide a plurality of icons fordisplay via at least one head-mounted output display, the plurality oficons configured for selection via at least one gesture tracking sensor;receive user gesture data via the at least one gesture tracking sensor,the user gesture data indicating selection of a selected icon of theplurality of icons; access a hierarchy of icons based on the selectionof the selected icon; and provide part of the hierarchy of icons fordisplay via the at least one head-mounted output display based on theselected icon. Clause 184. The medium of any clause mentioned in thisparagraph, wherein providing the plurality of icons for display furtheroccurs only when the at least one gesture tracking sensor senses an iconsummoning gesture. Clause 185. The medium of any clause mentioned inthis paragraph, wherein the icon summoning gesture comprises presentingan open hand for at least a threshold period of time. Clause 186. Themedium of any clause mentioned in this paragraph, wherein the pluralityof icons are presented using at least one cycle bubble animation. Clause187. The medium of any clause mentioned in this paragraph, wherein theicons are bubble icons. Clause 188. The medium of any clause mentionedin this paragraph, wherein the user gesture data comprises a relativehand motion hover-over pre-selection. Clause 189. The medium of anyclause mentioned in this paragraph, wherein the user gesture datacomprises a squeeze-to-move translation. Clause 190. The medium of anyclause mentioned in this paragraph, wherein the user gesture datacomprises a squeeze-to-pop selection. Clause 191. The medium of anyclause mentioned in this paragraph, wherein at least one of theplurality of icons includes an application logo. Clause 192. The mediumof any clause mentioned in this paragraph, wherein provide part of thehierarchy of icons for display via the at least one head-mounted outputdisplay includes: display at least one animation of the selected icon inresponse to the user gesture data. Clause 193. The medium of any clausementioned in this paragraph, wherein the at least one animationcomprises an inward/rolodex cycle. Clause 194. The medium of any clausementioned in this paragraph, wherein: the hierarchy of icons include afolder level and an application/file level. Clause 195. The medium ofany clause mentioned in this paragraph, wherein: the hierarchy of iconsincludes a settings level. Clause 196. The medium of any clausementioned in this paragraph, wherein: the hierarchy of icons furtherincludes a settings sublevel associated with at least one displayparameter. Clause 197.The medium of any clause mentioned in thisparagraph, wherein the at least one display parameter comprises acontinuous display parameter, and further comprising: receive selectionof the at least one settings sublevel icon associated with the at leastone display parameter via the at least one gesture tracking sensor;determine a relative spatial position of the settings sublevel icon andat least one user body party; provide via the output display aninterface element comprising a line stretching from the settingssublevel icon to the user body part; and adjust the at least one displayparameter and a length of the line based on the relative spatialposition. Clause (198): The method of clause 115, where said icons areabakographs. Clause (199): The method of clause 115, where said iconsare derived from abakographs. Clause (200): The method of clause 115,where said icons are toposculptures. Clause (201): The method of clause115, where said icons each correspond to one trace of an abakographicexposure. Clause (202): The method of clause 115, where said icons areeach a bead on an abakograph. Clause (203): The method of clause 134,where said icon, when selected, is rendered movable along saidabakograph.

It may be appreciated that the assemblies and modules described abovemay be connected with each other as may be required to perform desiredfunctions and tasks that are within the scope of persons of skill in theart to make such combinations and permutations without having todescribe each and every one of them in explicit terms. There is noparticular assembly, or components, that are superior to any of theequivalents available to the art. There is no particular mode ofpracticing the disclosed subject matter that is superior to others, solong as the functions may be performed. It is believed that all thecrucial aspects of the disclosed subject matter have been provided inthis document. It is understood that the scope of all claimed subjectmatter is not limited to: (i) the dependent claims, (ii) the detaileddescription of the non-limiting embodiments, (iii) the summary, (iv) theabstract, and/or (v) the description provided outside of this document(that is, outside of the instant application as filed, as prosecuted,and/or as granted). It is understood, for the purposes of this document,that the phrase “includes” is equivalent to the word “comprising.” It isnoted that the foregoing has outlined the non-limiting embodiments(examples). The description is made for particular non-limitingembodiments (examples). It is understood that the non-limitingembodiments are merely illustrative as examples.

1. (canceled)
 2. An apparatus, comprising: one or more sensors; ahead-mounted output display; a processing assembly operatively coupledto the one or more sensors and the head mounted output display, whereinthe processing assembly is configured to execute program instructionsthat cause the processing assembly to: receive data from the one or moresensors; process the data; access or create a spatial coordinate systemcorresponding to the data; determine a location corresponding to areal-world object within the spatial coordinate system; provide avirtual object for display by the output display in an augmented-reality(AR) space, where the virtual object and the real-world object occupysubstantially the same location in the spatial coordinate system;determine an amount of interaction between a user of the apparatus andthe real-world object using the processed data; determine a manipulationof the AR space by the user of the apparatus based on the determinedamount of interaction; and display a result of the user manipulation ofAR space by the head-mounted output display, wherein the display resultis perceived by the user of the apparatus as their interaction with theAR space.
 3. The apparatus of claim 2, wherein the user manipulation ofthe AR space includes a user manipulation of the virtual object.
 4. Theapparatus of claim 3, wherein the display result of the usermanipulation of the AR space includes altering a perception of thevirtual object by the user in response to the amount of interaction. 5.The apparatus of claim 2, wherein determining an amount of interactionincludes determining an amount of taction between the user and thereal-world object.
 6. The apparatus of claim 5, wherein the amount oftaction between the user and the real-world object includes determiningthe amount of taction between an appendage of the user and thereal-world object.
 7. The apparatus of claim 5, wherein determining theamount of taction includes processing a first signal derived from thedata indicating contact with the real-world object and a second signalderived from the data indicating within the spatial coordinate system acorresponding contact with a point, a portion, or an area of thereal-world object or the virtual object.
 8. The apparatus of claim 5,wherein determining the amount of taction includes determining one ofone or more levels of touching of the real-world object and the displayresult of the user manipulation of the AR space depends on the onedetermined level of touching.
 9. The apparatus of claim 5, whereinprocessing the data includes providing feature vectors to a neuralnetwork, where the neural network provides an output used to determine adegree of taction, and determining the manipulation is based on thedetermined degree of taction.
 10. The apparatus of claim 6, whereindetermining an amount of interaction includes determining a direction oftravel of the user appendage relative to a surface of the real-worldobject within the spatial coordinate system from the processed data, andthe determined manipulation is based at least on the determineddirection of travel.
 11. The apparatus of claim 6, wherein the data isprocessed to provide a depth map of the environment including thereal-world object and determine spatial coordinates, positions, ororientations of the user appendage.
 12. The apparatus of claim 11,wherein determining an amount of interaction between a user of theapparatus and the real-world object includes estimating a degree ofhomographic intrusion between the appendage and the real-world object.13. A method implemented by an apparatus including one or more sensors,a head mounted display, and a processing assembly, the methodcomprising: receiving data from the one or more sensors; processing thedata by the processing assembly; accessing or creating, by theprocessing assembly, a spatial coordinate system corresponding to thedata; determining, by the processing assembly, a location correspondingto a real-world object within the spatial coordinate system; providing avirtual object for display by the output display in an augmented-reality(AR) space, where the virtual object and the real-world object occupysubstantially the same location in the spatial coordinate system;determining, by the processing assembly, an amount of interactionbetween a user of the apparatus and the real-world object using theprocessed data; determining, by the processing assembly, a manipulationof the AR space by the user of the apparatus based on the determinedamount of interaction; and displaying a result of the user manipulationof AR space by the head-mounted output display, wherein the displayresult is perceived by the user of the apparatus as their interactionwith the AR space.
 14. The method of claim 13, wherein displaying aresult of the user manipulation of the AR space includes displaying auser manipulation of the virtual object.
 15. The method of claim 14,wherein the displayed result of the user manipulation of the AR spaceincludes altering a perception of the virtual object by the user inresponse to the amount of interaction.
 16. The method of claim 13,wherein determining an amount of interaction includes determining anamount of taction between the user and the real-world object.
 17. Themethod of claim 16, wherein determining the amount of taction betweenthe user and the real-world object includes determining the amount oftaction between an appendage of the user and the real-world object. 18.The method of claim 16, wherein determining the amount of tactionincludes processing a first signal derived from the data indicatingcontact with the real-world object and a second signal derived from thedata indicating within the spatial coordinate system a correspondingcontact with a point, a portion, or an area of the real-world object orthe virtual object.
 19. The method of claim 16, wherein determining theamount of taction includes determining one of one or more levels oftouching of the real-world object and the display result of the usermanipulation of the AR space depends on the one determined level oftouching.
 20. The method of claim 16, wherein processing the dataincludes providing feature vectors to a neural network, where the neuralnetwork provides an output used to determine a degree of taction, anddetermining the manipulation is based on the determined degree oftaction.
 21. The method of claim 17, wherein determining an amount ofinteraction includes determining a direction of travel of the userappendage relative to a surface of the real-world object within thespatial coordinate system from the processed data, and the determinedmanipulation is based at least on the determined direction of travel.22. The method of claim 17, wherein processing the data includesprocessing the data to provide a depth map of the environment includingthe real-world object and determining spatial coordinates, positions, ororientations of the user appendage.
 23. The method of claim 22, whereindetermining an amount of interaction between a user of the apparatus andthe real-world object includes estimating a degree of homographicintrusion between the appendage and the real-world object.